Massive file management method, device, computer equipment and readable storage medium

Abstract

The present application proposes a method, device, computer equipment and readable storage medium for managing massive files, wherein the method includes: determining the type of the file to be uploaded, extracting the characteristic information of the file to be uploaded according to the type; sending the file to be uploaded to FastDFS for storage, and obtaining the FileID returned by FastDFS; sending the characteristic information and the FileID to ElasticSearch for storage; when receiving a user's search instruction, searching in ElasticSearch according to the search conditions input by the user, and obtaining the target file FileID that meets the search conditions; querying in FastDFS according to the target file FileID, and obtaining the corresponding target file. The massive file management method of the present application stores the characteristic information and file storage information of the file separately, and when searching for files, obtains the storage information of the file according to the user's search conditions, thereby realizing a rapid search of massive files, especially semi-structured and unstructured format files.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method, apparatus, computer device and readable storage medium for managing massive files. Background Art

[0002] Internet technology is developing rapidly, and the information generated is growing explosively. Traditional file systems are unable to meet the needs of dynamic expansion and efficient access to massive files. Especially for semi-structured data or unstructured data, the space occupied by such data is usually larger than that of structured data, and the requirements for storage systems are higher. In addition, in some application scenarios, it is necessary to quickly search for semi-structured data or unstructured data from multiple flexible dimensions, which also puts forward requirements for search engines. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, computer equipment and readable storage medium for managing massive files, aiming to solve the problem of how to quickly search massive semi-structured or unstructured data.

[0004] In order to achieve the above-mentioned invention object, the present application proposes a method for managing massive files, including:

[0005] Determine the type of the file to be uploaded, and extract characteristic information of the file to be uploaded according to the type;

[0006] Send the file to be uploaded to FastDFS for storage and obtain the FileID returned by FastDFS;

[0007] Send the feature information and the FileID to ElasticSearch for storage;

[0008] When receiving a search instruction from a user, search in ElasticSearch according to the search conditions input by the user to obtain the target file FileID that meets the search conditions;

[0009] According to the target file FileID, query is performed in FastDFS to obtain the corresponding target file.

[0010] Furthermore, the step of determining the type of the file to be uploaded and extracting characteristic information of the file to be uploaded according to the type includes:

[0011] Determine the type of file to be uploaded;

[0012] When the type is a text file, the file name, file format and text content to be uploaded are extracted.

[0013] Furthermore, after the step of determining the type of the file to be uploaded, the step further includes:

[0014] When the type is a picture file, the file name, file format, color attribute value and image texture of the file to be uploaded are extracted.

[0015] Furthermore, the search condition is to search for similar images based on the uploaded image, and the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition includes:

[0016] Extracting color attribute values ​​and image texture of the uploaded image;

[0017] Calculate the Hamming distance between the image texture of the uploaded image and the stored image, and obtain the target file FileID whose Hamming distance meets the preset threshold;

[0018] The target file FileIDs are sorted according to the Hamming distances, and the target file FileIDs with the same Hamming distances are sorted again according to the color attribute values.

[0019] Furthermore, after the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition, the following steps are further included:

[0020] The search conditions input by the user are recorded, and the characteristic information corresponding to the search conditions whose search times are greater than a preset number are recorded as associated characteristic information.

[0021] Furthermore, the step of sending the file to be uploaded to FastDFS for storage also includes:

[0022] Query whether there is an associated file in the files to be uploaded whose file size is smaller than a preset threshold and whose characteristic information contains the associated characteristic information;

[0023] If so, the associated file is sent to FastDFS for merged storage.

[0024] The present application also provides a mass file management device, including:

[0025] A determination and extraction unit, used for determining the type of the file to be uploaded, and extracting characteristic information of the file to be uploaded according to the type;

[0026] The first sending unit is used to send the file to be uploaded to FastDFS for storage, and obtain the FileID returned by FastDFS;

[0027] A second sending unit, used for sending the characteristic information and the FileID to ElasticSearch for storage;

[0028] A search unit, used to search in ElasticSearch according to the search conditions input by the user, and obtain the target file FileID that meets the search conditions;

[0029] The query unit is used to query in FastDFS according to the target file FileID to obtain the corresponding target file.

[0030] Furthermore, the judgment and extraction unit includes:

[0031] The first execution module is used to extract the name, file format and text content of the file to be uploaded when the type is a text file.

[0032] The second execution module is used to extract the file name, file format, color attribute value and image texture of the file to be uploaded when the type is a picture file.

[0033] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0034] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0035] The massive file management method, device, computer equipment and readable storage medium of the present application, the massive file management method of the present application stores the feature information and file storage information of the file separately, and when searching for files, obtains the storage information of the file according to the user's search conditions, thereby realizing rapid search of massive files, especially semi-structured and unstructured format files. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method for managing a large number of files according to an embodiment of the present application;

[0037] Figure 2 This is a schematic structural diagram of a method and device for managing massive files according to an embodiment of the present application;

[0038] Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0039] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] Reference Figure 1 In an embodiment of the present application, a method for managing a large number of files is provided, comprising the steps of:

[0042] S1. Determine the type of the file to be uploaded, and extract characteristic information of the file to be uploaded according to the type;

[0043] S2. Send the file to be uploaded to FastDFS for storage, and obtain the FileID returned by FastDFS;

[0044] S3. Send the feature information and the FileID to ElasticSearch for storage.

[0045] S4. When receiving a search instruction from the user, search in ElasticSearch according to the search conditions input by the user to obtain the target file FileID that meets the search conditions;

[0046] S5. Query in FastDFS according to the target file FileID to obtain the corresponding target file.

[0047] Both FastDFS and ElasticSearch are distributed file systems, which are file systems that achieve goals such as file storage or big data computing through the collaboration of multiple computers. In this solution, the characteristic information in the file is extracted, FastDFS is used to store files, which can include structured data and unstructured data, and ElasticSearch is used to store the characteristic information and FileID of the file, that is, all data stored in ElasticSearch is structured data. FastDFS is suitable for storing unstructured data such as documents, pictures, audio, and video, and is especially suitable for storing massive small and medium-sized files. ElasticSearch can store, search, and analyze large amounts of data in a short period of time, and is suitable for complex search scenarios.

[0048] As described in the above step S1, the attribute module reads the file to be uploaded and determines the file type, and extracts the feature information of the file using a preset method according to the file type. For example, when the file type is a text file, the feature information may include text content, etc.; when the file type is a picture file, the feature information may include color attribute values ​​and image texture, etc.; when the file type is an audio file, the feature information may include zero-crossing rate, short-time energy, resonance peak, spectrogram, etc.; when the file type is a video file, the feature information may include key frames, etc.;

[0049] As described in step S2 above, FastDFS contains two roles: TrackerServer and StorageServer, which are responsible for scheduling tasks and storing file data respectively. After FastDFS receives a storage request, TrackerServer allocates the IP and port of an available StorageServer to the file. After the file is uploaded, StorageServer returns a FileID for unique identification. The FileID contains information including the IP of the source StorageServer, the file creation timestamp, the file size, the file CRC32 checksum, and a random number.

[0050] As described in step S3 above, before using ElasticSearch to save data, create an index library in advance, set the word segmenter, perform mapping configuration, and describe the results returned when creating the index. Send the extracted feature information and the returned FileID to the ElasticSearch that has created the index, perform word segmentation, store it in the mapping, create an index, and associate the feature information with the corresponding FileID and save it.

[0051] As described in step S4 above, according to the search conditions input by the user, the created index is searched in ElasticSearch, and the target file FileID that meets the search conditions is returned. Search can be divided into precise search and fuzzy search. For precise search, ElasticSearch will not segment the search terms, nor perform scoring or relevance calculation. According to the search conditions, filters are used to find matching data in the inverted index, and the FileID of the file containing the search field is obtained. ElasticSearch-SQL can be used for search. The fields of precise search can include, for example, dates, numbers, authors, etc., and common search conditions can be, for example, files with timestamps within the past hour, files of specified authors, etc. For fuzzy search, ElasticSearch performs lexical analysis, segmentation, and inverted index query on the statements (query) in the search conditions, scores the relevance according to the degree of matching, queries the target file FileID that meets the search conditions according to the score, and returns after sorting according to the relevance. Common search conditions in fuzzy retrieval can be, for example, querying files containing specified words. In a single search condition, precise search and fuzzy search can be included at the same time.

[0052] As described in step S5 above, a query is performed in FastDFS according to the target file FileID, and the StorageServer locates the target file according to the target file FileID. In some implementations, the corresponding target file is obtained in FastDFS in a paging manner according to the target file FileID, and sent to the client in batches.

[0053] In one embodiment, the step S1 of determining the type of the file to be uploaded and extracting characteristic information of the file to be uploaded according to the type includes:

[0054] S11, determining the type of the file to be uploaded;

[0055] S12. When the file type is a text file, extract the file name, file format, and text content to be uploaded.

[0056] As described in step S11 above, the file type can be determined based on the file suffix, or based on the characteristics of some bytes at the beginning of the file. As described in step S12 above, for long text in the text content, such as the main body of an email or the body of a document, the text content is split into separate terms (Term) by a word segmenter in ElasticSearch, and an inverted index is established for the obtained terms. For keywords in the text content, such as data such as an email address, name, document tag, etc., word segmentation is not required, and the index is directly established after extraction. Based on the FileID of the file and the above-extracted feature information, the supported search conditions may include: file upload time (file creation timestamp), file size, file name, file format, keywords, etc. Users can query by a single search condition or by a joint query by multiple conditions. For example, when the search condition is an accurate file name, a unique target file can be queried; when the search condition is a certain file format, all file sets of this specific format can be queried; when the search condition includes files with a specified upload time and a specified phrase, a file set uploaded at a specified time and containing a specified phrase sorted by relevance can be queried.

[0057] In one embodiment, after the step S11 of determining the type of the file to be uploaded, the method further includes:

[0058] S13. When the type is a picture file, extract the file name, file format, color attribute value and image texture of the file to be uploaded.

[0059] As mentioned above, for image files, color is a global feature, which is a feature based on pixels. Color features are invariant to translation and rotation and are insensitive to various deformations. The specific performance of color attribute values ​​depends on the color model used. Common color models include RGB model and HSV model. In a specific implementation, the RGB model is implemented using the PIL module of python. First, a preset number of pixels are extracted to obtain a scaled image of the image, and then the values ​​of the three primary colors of R, G, and B, r, g, and b, are extracted. Then, the RGB color model is converted to the HSV color model through this result. The extraction of color attribute values ​​enables the system to search for images through the content features of the image. Image texture can depict the inherent laws and structures of the surface of an object. Its basic principle is to use the grayscale value of the pixel to find the law of pixel distribution in the image. It is a statistical feature with rotation invariance and strong resistance to noise. The image texture extraction method can use the average hash algorithm (aHash), the perceptual hash algorithm (pHash) or the difference hash algorithm (dHash) to convert the extracted features into a certain number of binary code sequences, i.e., hash coding. The extraction of image texture can realize the search of similar images. Therefore, based on the file's FileID and the above-extracted feature information, supported search conditions may include: file upload time (file creation timestamp), file size, file name, file format, image color, similar image search, etc.

[0060] In one embodiment, the search condition described in the above step S4 is to search for similar images based on the uploaded image, and the step of searching in ElasticSearch according to the search condition input by the user and obtaining the target file FileID that meets the search condition includes:

[0061] S41, extracting the color attribute value and image texture of the uploaded image;

[0062] S42, calculating the Hamming distance between the image texture of the uploaded image and the image stored, and obtaining the target file FileID whose Hamming distance meets the preset threshold;

[0063] S43, sorting the target file FileIDs according to the Hamming distances, and sorting the target file FileIDs with the same Hamming distances again according to the color attribute values.

[0064] As described in step S41 above, after receiving the picture uploaded by the user, the color attribute value and image texture of the picture are calculated. In a specific embodiment, the color attribute value includes the RGB value and HSV value of the picture. As described in step S42 above, the Hamming distance (Hamming Distance) between the uploaded picture and the image texture of the stored picture is calculated. The Hamming distance is the number of different characters in the corresponding positions of two equal-length strings. For binary strings, it is the number of characters that need to be replaced to convert one set of binary strings into another set of binary strings. For example: the Hamming distance between 1011101 and 1001001 is 2; the Hamming distance between 2143896 and 2233796 is 3. The smaller the Hamming distance, the higher the similarity of the pictures. According to the calculation result, the target file FileID whose Hamming distance meets the preset threshold is obtained. In a specific embodiment, the preset threshold is that the Hamming distance is less than 5, that is, the target file FileID whose Hamming distance is less than 5 is obtained. As described in step S43 above, the target file FileID is sorted by similarity according to the Hamming distance, and the file with smaller Hamming distance is placed in front, so as to preferentially search for target files with higher similarity in FastDFS and return them to the user. In the similarity sorting, for target file FileIDs with the same Hamming distance, they are sorted again according to the color attribute value of the uploaded image. Specifically, the similarity of the image can be evaluated by calculating the Minkowsky distance or the quadratic distance, thereby completing the re-sorting of the target file FileID.

[0065] In one embodiment, the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition further includes:

[0066] S6. Record the search conditions input by the user, and record the characteristic information corresponding to the search conditions whose search times are greater than a preset number as associated characteristic information.

[0067] In a specific embodiment, the step S2 of sending the file to be uploaded to FastDFS for storage further includes:

[0068] S21, querying whether there is an associated file in the files to be uploaded whose file size is smaller than a preset threshold and whose characteristic information contains the associated characteristic information;

[0069] S22: If so, send the associated file to FastDFS for merged storage.

[0070] In FastDFS, when the number of small files stored is very large, a large number of small files will occupy the inode resources of the storage system, and the number of inodes is limited, which may lead to the situation that the storage server is unable to provide storage services due to the exhaustion of inodes when there is still storage space, and the storage space cannot be fully utilized. In addition, the efficiency of the system locating small files will also be sharply reduced. FastDFS can merge and store multiple small files into a larger Trunk file, and access the small files according to the FileID of the returned large file and the offset of the small file. The large file created by default is usually 64MB, the minimum value of the small file is 256 bytes, and the maximum value is 16MB. The preset threshold in this scheme can be between 256 bytes and 16MB. According to the recorded search conditions, this scheme records the characteristic information corresponding to the commonly used search conditions (the number of searches is greater than the preset number) as associated characteristic information, defines multiple files with associated characteristic information in the files to be uploaded as associated files, and merges and stores the associated files. In other words, the files that are often searched in batches are merged and stored to make full use of the storage space and improve the efficiency of reading files. In a specific implementation, by analyzing the recorded historical search conditions, it is found that the search conditions with the number of searches greater than the preset number are: the file name contains "coffee" and the file format is html. The characteristic information corresponding to this search condition is two characteristic information, namely, the file name contains "coffee" and the file format is html. Based on this, the two characteristic information, namely, the file name contains "coffee" and the file format is html, is recorded as a set of associated characteristic information. It should be noted that, depending on the search condition, a set of associated characteristic information may include one characteristic information or multiple characteristic information. The system can record multiple sets of associated characteristic information. For multiple sets of associated characteristic information, their relevance can be sorted according to the number of searches, and a higher number of searches corresponds to a higher relevance. The sorting can be regularly updated according to the number of historical searches. Multiple files with a set of associated characteristic information are defined as associated files. In each subsequent storage process, after extracting the characteristic information of the file to be uploaded, multiple files to be uploaded whose file size is less than a preset threshold are screened out, and multiple files to be uploaded whose characteristic information contains the associated characteristic information are further screened out, i.e., associated files, and the associated files are merged and stored in FastDFS. When the system records multiple sets of associated characteristic information, the associated files can be screened and merged and stored in sequence according to the sorting of the relevance.

[0071] Reference Figure 2 In an embodiment of the present application, a device for managing a large number of files is also provided, including:

[0072] The judging and extracting unit 10 is used to judge the type of the file to be uploaded, and extract characteristic information of the file to be uploaded according to the type;

[0073] The first sending unit 20 is used to send the file to be uploaded to FastDFS for storage, and obtain the FileID returned by FastDFS;

[0074] A second sending unit 30 is used to send the feature information and the FileID to ElasticSearch for storage;

[0075] The search unit 40 is used to search in ElasticSearch according to the search conditions input by the user when receiving the search instruction from the user, and obtain the target file FileID that meets the search conditions;

[0076] The query unit 50 is used to query in FastDFS according to the target file FileID to obtain the corresponding target file.

[0077] In a specific embodiment, the judgment and extraction unit 10 includes:

[0078] The first execution module is used to extract the name, file format and text content of the file to be uploaded when the type is a text file.

[0079] The second execution module is used to extract the file name, file format, color attribute value and image texture of the file to be uploaded when the type is a picture file.

[0080] The components of the device can realize the functions of any one of the methods for managing massive files as described above, and the specific structure will not be described in detail.

[0081] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as pictures, documents, audio and video. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for managing massive files is implemented.

[0082] The above-mentioned processor executes the above-mentioned method for managing massive files, including: sending the file to be uploaded to FastDFS for storage, and obtaining the FileID returned by FastDFS; sending the characteristic information and the FileID to ElasticSearch for storage; when receiving a user's search instruction, searching in ElasticSearch according to the search conditions entered by the user, and obtaining the target file FileID that meets the search conditions; querying in FastDFS according to the target file FileID to obtain the corresponding target file.

[0083] In one embodiment, the step of determining the type of the file to be uploaded and extracting characteristic information of the file to be uploaded according to the type includes: determining the type of the file to be uploaded; when the type is a text file, extracting the file name, file format and text content to be uploaded.

[0084] In a specific embodiment, after the step of determining the type of the file to be uploaded, the step further includes: when the type is a picture file, extracting the file name, file format, color attribute value and image texture of the file to be uploaded.

[0085] In one embodiment, the search condition is to search for similar images based on the uploaded image, and the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition includes: extracting the color attribute value and image texture of the uploaded image; calculating the Hamming distance between the uploaded image and the image texture of the stored image, and obtaining the target file FileID whose Hamming distance meets a preset threshold; sorting the target file FileID according to the Hamming distance, and sorting the target file FileID with the same Hamming distance again according to the color attribute value.

[0086] In one embodiment, the step of searching in ElasticSearch according to the search conditions input by the user and obtaining the target file FileID that meets the search conditions also includes: recording the search conditions input by the user, and recording the feature information corresponding to the search conditions that have been searched more than a preset number of times as associated feature information.

[0087] In one embodiment, the step of sending the file to be uploaded to FastDFS for storage also includes: querying whether there is an associated file in the file to be uploaded whose file size is smaller than a preset threshold and whose characteristic information contains the associated characteristic information; if so, sending the associated file to FastDFS for merged storage.

[0088] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for managing massive files is implemented, including: sending the file to be uploaded to FastDFS for storage, and obtaining the FileID returned by FastDFS; sending the feature information and the FileID to ElasticSearch for storage; when receiving a user's search instruction, searching in ElasticSearch according to the search conditions input by the user, and obtaining the target file FileID that meets the search conditions; querying in FastDFS according to the target file FileID, and obtaining the corresponding target file.

[0089] In one embodiment, the step of determining the type of the file to be uploaded and extracting characteristic information of the file to be uploaded according to the type includes: determining the type of the file to be uploaded; when the type is a text file, extracting the file name, file format and text content to be uploaded.

[0090] In a specific embodiment, after the step of determining the type of the file to be uploaded, the step further includes: when the type is a picture file, extracting the file name, file format, color attribute value and image texture of the file to be uploaded.

[0091] In one embodiment, the search condition is to search for similar images based on the uploaded image, and the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition includes: extracting the color attribute value and image texture of the uploaded image; calculating the Hamming distance between the uploaded image and the image texture of the stored image, and obtaining the target file FileID whose Hamming distance meets a preset threshold; sorting the target file FileID according to the Hamming distance, and sorting the target file FileID with the same Hamming distance again according to the color attribute value.

[0092] In one embodiment, the step of searching in ElasticSearch according to the search conditions input by the user and obtaining the target file FileID that meets the search conditions also includes: recording the search conditions input by the user, and recording the feature information corresponding to the search conditions that have been searched more than a preset number of times as associated feature information.

[0093] In one embodiment, the step of sending the file to be uploaded to FastDFS for storage also includes: querying whether there is an associated file in the file to be uploaded whose file size is smaller than a preset threshold and whose characteristic information contains the associated characteristic information; if so, sending the associated file to FastDFS for merged storage.

[0094] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0095] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0096] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for managing massive files, characterized in that: include: Determine the type of the file to be uploaded, and extract characteristic information of the file to be uploaded according to the type; Send the file to be uploaded to FastDFS for storage and obtain the FileID returned by FastDFS; Send the feature information and the FileID to ElasticSearch for storage; When receiving a search instruction from a user, search in ElasticSearch according to the search conditions input by the user to obtain the target file FileID that meets the search conditions; According to the target file FileID, query in FastDFS to obtain the corresponding target file; The step of determining the type of the file to be uploaded and extracting characteristic information of the file to be uploaded according to the type comprises: Determine the type of file to be uploaded; When the type is a text file, extract the file name, file format and text content to be uploaded; After the step of determining the type of the file to be uploaded, the method further includes: When the type is a picture file, extract the file name, file format, color attribute value and image texture of the file to be uploaded; The search condition is to search for similar images based on the uploaded image. The step of searching in ElasticSearch according to the search condition input by the user and obtaining the target file FileID that meets the search condition includes: Extracting color attribute values ​​and image texture of the uploaded image; Calculate the Hamming distance between the image texture of the uploaded image and the stored image, and obtain the target file FileID whose Hamming distance meets the preset threshold; The target file FileIDs are sorted according to the Hamming distances, and the target file FileIDs with the same Hamming distances are sorted again according to the color attribute values.

2. The method for managing massive files according to claim 1, characterized in that: After the step of searching in ElasticSearch according to the search condition input by the user to obtain the target file FileID that meets the search condition, the following steps are further included: The search conditions input by the user are recorded, and the characteristic information corresponding to the search conditions whose search times are greater than a preset number are recorded as associated characteristic information.

3. The method for managing massive files according to claim 2, characterized in that: The step of sending the file to be uploaded to FastDFS for storage also includes: Query whether there is an associated file in the files to be uploaded whose file size is smaller than a preset threshold and whose characteristic information contains the associated characteristic information; If so, the associated file is sent to FastDFS for merged storage.

4. A mass file management device, characterized in that: A method for managing a large number of files according to any one of claims 1 to 3, comprising: A determination and extraction unit, used for determining the type of the file to be uploaded, and extracting characteristic information of the file to be uploaded according to the type; The first sending unit is used to send the file to be uploaded to FastDFS for storage, and obtain the FileID returned by FastDFS; A second sending unit, used for sending the characteristic information and the FileID to ElasticSearch for storage; A search unit, used to search in ElasticSearch according to the search conditions input by the user, and obtain the target file FileID that meets the search conditions; The query unit is used to query in FastDFS according to the target file FileID to obtain the corresponding target file.

5. The mass file management device according to claim 4, characterized in that: The judgment and extraction unit comprises: A first execution module, for extracting the name, file format and text content of the file to be uploaded when the type is a text file; The second execution module is used to extract the file name, file format, color attribute value and image texture of the file to be uploaded when the type is a picture file.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images