File management and retrieval method, apparatus and device, and computer program product

Abstract

The invention discloses a file management and retrieval method, device and equipment and a computer program product, the method is applied to a storage server, and the method comprises the following steps: receiving a file uploaded by a client; storing the file in a file system of a storage server and obtaining basic index information of the file; determining whether the file meets a first hotspot index condition or not; if yes, establishing deep index information of the file, and storing the basic index information of the file and the deep index information of the file into a preset index structure; and performing file retrieval according to the index information of the file system or the index information of the preset index structure. According to the method, only the necessary files are subjected to hotspot data indexing, and the index information of the file system is fully utilized, so that the calculation overhead caused by full-amount file indexing is avoided. Through a hierarchical storage strategy, the overall resource utilization rate of the storage device is improved, and a client user is supported to achieve rapid file retrieval and operation on a limited device.

Description

Technical Field

[0001] This application relates to the field of document management technology, and in particular to a document management and retrieval method, apparatus and equipment, and computer program product. Background Technology

[0002] In home settings, devices such as routers and set-top boxes are often used as storage service devices. These devices provide data storage services to users through built-in storage or external storage expansion (such as USB flash drives and portable hard drives). However, due to cost and design limitations, these devices typically have lower CPU and memory resources, making it difficult to support complex file indexing and management functions, such as full file metadata extraction or efficient full-text search mechanisms.

[0003] Traditional file management systems typically rely on high-performance hardware devices (such as PCs, NAS devices, or cloud services). These devices, with their powerful computing resources, can fully index all files and build complex retrieval structures for rapid file classification, searching, and management. However, this approach usually depends on separate indexing services or object storage technologies, rarely utilizing the structured storage information of the file system itself, treating files as opaque storage objects. While this design works well on high-performance devices, for devices with limited computing resources (such as routers and set-top boxes), over-reliance on full indexing and complex retrieval structures leads to significant performance and storage overhead, making it difficult to operate efficiently in resource-constrained environments.

[0004] Therefore, how to achieve efficient file management and retrieval functions on storage devices with limited computing resources, while reducing the consumption of computing and storage resources, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a file management and retrieval method, apparatus, device, and computer program product to reduce resource consumption and demands on the server and improve file management and retrieval efficiency.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a file management and retrieval method, applied to a storage service client, the file management and retrieval method comprising:

[0008] Receive files uploaded by the client;

[0009] The file is stored in the file system of the storage service client, and the basic index information of the file is obtained;

[0010] Determine whether the file meets the first hotspot indexing condition;

[0011] If the file meets the first hotspot indexing condition, a deep index information for the file is established, and the basic index information and the deep index information of the file are stored in a preset index structure.

[0012] File retrieval is performed based on the index information of the file system or the index information of the preset index structure.

[0013] Optionally, determining whether the file satisfies the first hotspot indexing condition includes:

[0014] Obtain hotspot file configuration information, wherein the hotspot file configuration information includes at least one of the types and ranges of hotspot files;

[0015] Based on the hotspot file configuration information, determine whether the file meets the first hotspot index condition.

[0016] Optionally, the step of retrieving files based on the index information of the file system or the index information of the preset index structure includes:

[0017] Receive file query requests from users;

[0018] In response to a user's file query request, determine whether the file corresponding to the file query request can be found in the cache;

[0019] If possible, the cached file query results will be returned to the client.

[0020] If not, then file retrieval is performed in the file system's index information or the index information of the preset index structure according to the file query request.

[0021] Optionally, the step of performing file retrieval based on the file query request in the index information of the file system or the index information of the preset index structure includes:

[0022] In response to the file query request, determine whether the file to be queried is a hot file;

[0023] If so, then according to the file query request, a query is performed in the index information of the preset index structure to obtain the first file query result and return it to the client, and the first file query result is stored in the cache;

[0024] If not, then according to the file query request, a query is performed in the index information of the file system to obtain the second file query result and return it to the client, and the second file query result is stored in the cache.

[0025] Optionally, the file management and retrieval method further includes:

[0026] Determine whether to trigger the cache invalidation mechanism based on file changes in the file system;

[0027] If the cache invalidation mechanism is triggered, invalid data in the cache is identified and cleaned up.

[0028] Optionally, the file management and retrieval method further includes:

[0029] The query data of the file is statistically analyzed based on the file query results in the cache.

[0030] Determine whether the file meets the second hotspot indexing condition based on the queried data of the file;

[0031] If the file satisfies the second hotspot indexing condition, a deep index information for the file is established, and the basic index information and the deep index information of the file are added to the preset index structure.

[0032] Optionally, the file management and retrieval method further includes:

[0033] Determine whether the hotspot index update condition has been triggered;

[0034] When the hotspot index update condition is triggered, the index information in the preset index structure is updated.

[0035] Optionally, the condition for determining whether to trigger a hotspot index update includes:

[0036] Whether to trigger the hotspot index update condition is determined based on at least one of the following: file operation records, storage status of the preset index structure, and hotspot file configuration adjustment information.

[0037] If, based on the file operation records, it is determined that a file has been modified or has not been operated on for a long time, or the storage status of the preset index structure is saturated, or the hot file configuration adjustment information is received, then the conditions for triggering the hot index update are determined.

[0038] Otherwise, it is determined that the hotspot index update condition has not been triggered.

[0039] Secondly, embodiments of this application also provide a file management and retrieval device for use in a storage service terminal, the file management and retrieval device comprising:

[0040] The receiving unit is used to receive files uploaded by the client;

[0041] The first indexing unit is used to store the file in the file system of the storage service terminal and establish the basic index information of the file;

[0042] The first determining unit is used to determine whether the file satisfies the first hotspot index condition;

[0043] The second indexing unit is used to establish the deep index information of the file when the file meets the first hot spot indexing condition, and to store the basic index information and the deep index information of the file into a preset index structure.

[0044] The retrieval unit is used to perform file retrieval based on the index information of the file system or the index information of the preset index structure.

[0045] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0046] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned file management and retrieval methods.

[0047] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the aforementioned file management and retrieval methods.

[0048] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: The file management and retrieval method of this application embodiment is applied to the storage service terminal. First, it receives files uploaded by the client; then, it stores the files in the file system of the storage service terminal and obtains the basic index information of the files; then, it determines whether the files meet the first hot spot index condition; then, if the files meet the first hot spot index condition, it establishes the deep index information of the files, and stores the basic index information and the deep index information of the files in a preset index structure; finally, it performs file retrieval based on the index information of the file system or the index information of the preset index structure. The file management and retrieval method of this application embodiment only performs hot spot data indexing on necessary files and makes full use of the index information provided by the file system, avoiding the computational overhead caused by full file indexing. Through the hierarchical storage strategy, the overall resource utilization of the storage device is improved. It supports client users to achieve fast file retrieval and operation on limited devices, meets the daily usage needs of users, and can adapt to diverse usage scenarios. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 A flowchart illustrating the file management and retrieval methods;

[0051] Figure 2 This is a schematic diagram of a file upload process in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of a document query process in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of an index update process in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a file management and retrieval device according to an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0058] Currently, most existing file management and retrieval technologies rely on high-performance server-side hardware (such as NAS, dedicated storage servers, or cloud storage services) to perform full indexing and metadata extraction of files using high-performance computing resources, and to construct complex retrieval structures. While this design provides powerful file management and retrieval capabilities, it also has the following main problems:

[0059] (1) Strong hardware dependency: The construction of full indexes and complex retrieval structures requires a large amount of computing resources and storage space, resulting in a dependence on high-performance server hardware. For resource-constrained devices (such as home routers, set-top boxes, or edge devices), similar functions cannot be implemented.

[0060] (2) Large amount of index data: The full index design extracts metadata from all files, resulting in a large amount of index data, which is not suitable for scenarios with limited storage resources.

[0061] (3) Limited retrieval efficiency: When the file size is large or the access frequency is high, the full index design will reduce retrieval efficiency due to frequent complex query operations, which will affect the user experience.

[0062] (4) Insufficient scalability: Existing technologies lack the ability to flexibly adapt to user needs, such as dynamic management and customized support for hot file types, which cannot meet the specific needs of some users.

[0063] Based on this, embodiments of this application provide a file management and retrieval method, applied to a storage service client, such as... Figure 1 The diagram provided illustrates a flowchart of a file management and retrieval method according to an embodiment of this application. The file management and retrieval method includes at least the following steps S110 to S150:

[0064] Step S110: Receive the file uploaded by the client.

[0065] The file management and retrieval method of this application embodiment is applicable to resource-constrained storage service terminals (such as routers, set-top boxes, etc. in home scenarios). It optimizes the use of computing and storage resources through a hierarchical indexing strategy, while supporting efficient file retrieval.

[0066] Combination Figure 2 This application provides a schematic diagram of a file upload process according to an embodiment. First, the file uploaded by the client is received. The client device supports multiple devices (mobile phones, tablets, PCs, etc.) uploading files to the storage service provider via standard protocols (such as HTTP, FTP, SMB). File types can include ordinary files (such as documents, compressed files) and media files (such as images, videos, audio, etc.).

[0067] Step S120: Store the file in the file system of the storage service terminal and obtain the basic index information of the file.

[0068] Files uploaded by the client are written to the file system on the storage service server. The file system automatically allocates storage space and records metadata. The file system can store files in internal storage or external extended storage devices such as USB flash drives or external hard drives. Because the file system itself is a structured organization, when a file is stored in the file system, the relevant file information, such as file path, file name, file size, file creation time, and file modification time, constitutes the basic index information of that file, avoiding the need to perform a full index of all files.

[0069] Based on the directory structure of the file system, it provides fast basic search capabilities by simple keywords such as path and time, ensuring that basic file management needs are met with low resource consumption.

[0070] Step S130: Determine whether the file meets the first hotspot indexing condition.

[0071] The primary hotspot indexing criteria can be dynamically defined based on user configuration or preset rules. For example, users can configure the types and ranges of hot files, such as images, videos, and documents. The storage service client can maintain a hotspot configuration table, storing user-configured information such as the types and ranges of hot files. For newly uploaded files from the client, the system can match file types (such as extensions and path prefixes) against the configuration table to determine if the primary hotspot indexing criteria are met.

[0072] Step S140: If the file meets the first hotspot indexing condition, establish the deep index information of the file, and store the basic index information and the deep index information of the file into a preset index structure.

[0073] If the currently uploaded file meets the first hotspot indexing criteria, further build the file's deep index information. For example, extract structured data such as EXIF ​​data of images, keyframe timestamp data of videos, and titles and summaries of documents. Combine this with the basic index information of the file obtained in the previous steps to form the index data of the hotspot file.

[0074] This application employs a hierarchical storage strategy to store file index data separately from the original files. Specifically, the file index data, including basic index information and deep index information, is stored in an efficient data structure such as an embedded database, forming a lightweight hotspot index to support fast retrieval and display. Only necessary metadata is extracted from hotspot files and stored in the lightweight index structure, reducing system resource consumption. The original files are stored in the file system's built-in storage or external extended storage devices such as USB flash drives or external hard drives, and are associated with the index via file paths.

[0075] Step S150: Perform file retrieval based on the index information of the file system or the index information of the preset index structure.

[0076] During the file retrieval phase, hierarchical retrieval is performed based on the user's query conditions in the index information of the aforementioned file system or the index information of the preset index structure. For example, hot files are preferentially retrieved in the index information of the preset index structure, and quick location is achieved through the hot index in the database; non-hot files are directly retrieved through the index information of the file system, avoiding the computational burden caused by full indexing.

[0077] The file management and retrieval method in this application only performs hotspot data indexing on necessary files and makes full use of the index information built into the file system, avoiding the computational overhead of indexing all files. Through a hierarchical storage strategy, the overall resource utilization of the storage device is improved. It supports client users in achieving fast file retrieval and operation on limited devices, meeting users' daily usage needs and adapting to diverse usage scenarios.

[0078] In some embodiments of this application, determining whether the file meets the first hotspot indexing condition includes: obtaining hotspot file configuration information, the hotspot file configuration information including at least one of hotspot file type and range; and determining whether the file meets the first hotspot indexing condition based on the hotspot file configuration information.

[0079] In the embodiments of this application, the core logic for determining whether a file meets the first hotspot indexing condition can be to implement a user-defined hotspot indexing strategy through dynamic configuration, thereby flexibly adapting to file management needs in different scenarios. The specific implementation process is as follows:

[0080] The storage service allows client users to configure hotspot file rules through a visual interface (such as a web management page or a mobile app). For example, users can specify file extensions for which hotspot indexing should be built, such as .jpg, .mp4, .pdf, etc., or select predefined category tags such as "images," "videos," "documents," etc. Users can also configure the scope of hotspot files, for example, making it only effective for files in a specific directory (such as / Photos / 2023) or on a specific storage device, avoiding the resource consumption of global indexing. Configuration information can be stored on the server in a structured format and supports dynamic updates.

[0081] When a client uploads a new file, the server can obtain the file type (by extension or MIME type) and storage path from the file metadata. The server then compares the file attributes with the list of file types or the configured file range in the hotspot configuration information. For example, if the user has configured "all images and videos" as hotspot files, the server will match whether the file extension belongs to a predefined set of image formats such as .jpg, .png, or video formats such as .mp4, .avi, etc. If the currently uploaded file attributes meet any of the configuration rules, it is determined to meet the first hotspot index condition, and a deep index of the file is created; otherwise, only the basic index of the file is stored.

[0082] This application's embodiments support user-defined hotspot index conditions, avoiding the creation of a full index and significantly reducing index computation and storage overhead. Through the design of "user configuration-driven + dynamic rule matching," low-cost and highly flexible file index management is achieved on resource-constrained devices, balancing performance optimization and user experience, and is particularly suitable for dynamically changing scenarios such as home and small office environments.

[0083] In some embodiments of this application, the step of performing file retrieval based on the index information of the file system or the index information of the preset index structure includes: receiving a user's file query request; responding to the user's file query request, determining whether the file corresponding to the file query request can be found in the cache; if so, returning the file query result in the cache to the client; if not, performing file retrieval based on the file query request in the index information of the file system or the index information of the preset index structure.

[0084] Combination Figure 3 This application provides a schematic diagram of a file query process according to an embodiment. During the file retrieval stage, the storage service client can receive file query requests sent by the client through a network interface. The request may include query conditions such as filename keywords, file type, time range, and file size. The server performs validity checks on the request (such as parameter format verification and permission checks) to ensure that the query conditions conform to preset rules and to prevent invalid requests from triggering subsequent processing flows.

[0085] The storage service can maintain an in-memory cache pool to store recent query results and their corresponding query conditions. When a query request is received, the current query conditions are first compared with historical query conditions in the cache. If a complete match is found, the server directly retrieves the corresponding file list (including file paths, metadata, etc.) from the cache and returns it to the client, thus eliminating the need for index retrieval and improving retrieval efficiency. If no matching query conditions are found in the cache, the server can further dynamically select an index source based on the query conditions, such as retrieving information based on file system indexes or indexes based on a preset index structure.

[0086] This application's embodiments optimize file retrieval response speed and system resource consumption by introducing a caching mechanism and a dynamic index query mechanism. The caching mechanism reduces direct read operations on the file system and index database, lowering I / O pressure and CPU computational load. The combination of hierarchical indexing and caching mechanisms enables the system to flexibly support various query scenarios, meeting the personalized needs of multiple scenarios.

[0087] In some embodiments of this application, the step of performing file retrieval in the index information of the file system or the index information of the preset index structure according to the file query request includes: in response to the file query request, determining whether the file to be queried is a hot file; if so, querying in the index information of the preset index structure according to the file query request to obtain a first file query result and returning it to the client, and storing the first file query result in the cache; if not, querying in the index information of the file system according to the file query request to obtain a second file query result and returning it to the client, and storing the second file query result in the cache.

[0088] Continue to refer to Figure 3 If no results matching the user's query criteria are found in the cache, the storage service further parses information such as file type in the query criteria and combines this with the configured hot file configuration information to determine whether the file being queried is a hot file. For example, when querying "2024.mp4 file", if the hot file rule includes "type = video", then the file being queried is determined to be a hot file; if querying "2020.txt file", then the file being queried is determined to be a non-hot file, i.e., a regular file.

[0089] If a file is identified as a hot file, a pre-defined index structure, such as an embedded database, can be directly queried. This index pre-stores metadata information for hot files, such as path, shooting time, and keyword tags. It supports quickly locating files based on query conditions, thus obtaining the first file query result. The first file query result can include a list of files and detailed metadata, which is synchronously cached in memory after being returned to the client.

[0090] If the file is determined to be a non-hotspot file, the server can query the basic index information through the file system interface to obtain the second file query result. This query only relies on metadata natively maintained by the file system, such as path, filename, and modification time, without requiring additional index calculations. The second file query result is also cached after being returned to the client.

[0091] By storing index information for frequently accessed files using a pre-defined index structure, the query response time for these files is significantly improved. The tiered query strategy avoids the computational and storage overhead of indexing the entire file system, extending the stable operating time of resource-constrained devices.

[0092] In some embodiments of this application, the file management and retrieval method further includes: determining whether to trigger a cache invalidation mechanism based on file changes in the file system; and, if the cache invalidation mechanism is triggered, identifying and cleaning up invalidated data in the cache.

[0093] In this embodiment of the application, file change events can be monitored through the underlying interface provided by the file system. File change events may include file modification, deletion, and other events. These change operations will affect the file query results stored in the cache, and therefore can be considered as triggering the cache invalidation mechanism.

[0094] When the cache invalidation mechanism is triggered, the range of cached data that needs to be invalidated is dynamically determined based on the correlation between the changed file and the data in the cache. This cached data is then automatically cleaned up to ensure that the system reloads the latest data on the next query, thus ensuring the accuracy of the returned results. For example, if the file change is that a file has been deleted, the system can iterate through all query results in the cache based on the file path or unique identifier of that file and clean up the query cache containing that file.

[0095] By capturing file change operations in real time and precisely triggering the cache invalidation mechanism, strong consistency between query results and file system status, as well as the accuracy of retrieval results, are ensured.

[0096] In some embodiments of this application, the file management and retrieval method further includes: statistically analyzing the queried data of a file based on the file query results in the cache; determining whether the file satisfies a second hotspot index condition based on the queried data of the file; establishing a deep index information for the file if the file satisfies the second hotspot index condition; and adding the basic index information and the deep index information of the file to the preset index structure.

[0097] Continue to refer to Figure 3 As mentioned earlier, the file search results are stored in the cache. Therefore, the cache is equivalent to retaining the file query records within a certain period of time. By performing statistical analysis on the file query results in the cache, it is possible to analyze which files(s) have been frequently queried by the user. If the number of queries or query frequency of a certain file exceeds a certain threshold, it means that the file belongs to the file that the user has been paying close attention to recently, which triggers the second hot spot index condition. That is, although the file was not classified as a hot file based on the user's previous configuration, it can now be marked as a hot file based on the user's query behavior, and the deep index information of the file can be constructed and added to the preset index structure together with the basic index information.

[0098] By dynamically monitoring query data in the cache, we can further capture the hot files that users are interested in based on user behavior, improve the flexibility of building hot file index information, and further meet users' search needs.

[0099] In some embodiments of this application, the file management and retrieval method further includes: determining whether a hotspot index update condition is triggered; and updating the index information in the preset index structure if the hotspot index update condition is triggered.

[0100] Combination Figure 4 This application provides a schematic diagram of an index update process according to an embodiment of the present application. This embodiment supports dynamic management of index information in a preset index structure, and can dynamically monitor whether hotspot index update conditions are triggered. Hotspot index update conditions may include, for example, conditions based on file operation records, the storage status of the preset index structure, and hotspot file configuration adjustment information. For instance, index information can be updated based on user file operations, certain index records can be deleted based on the storage status of the preset index structure, and index strategies and the type and scope of hotspot data can be dynamically adjusted according to user needs, thereby adapting to diverse usage scenarios.

[0101] All of the above situations can be considered as triggering hotspot index update conditions, and the index information in the preset index structure should be updated accordingly based on the specific trigger type. For example, when the file content or metadata changes, only the affected index portion can be updated to avoid a full reindex; when a file is deleted, the corresponding hotspot index records are cleaned up simultaneously to maintain the simplicity and consistency of the index structure.

[0102] This application embodiment achieves efficient utilization of computing resources through dynamic index management, reduces the storage occupation of index records, improves the overall utilization efficiency of storage devices, and supports users to dynamically adjust hotspot configuration information according to their needs, adapting to diverse usage scenarios.

[0103] In some embodiments of this application, determining whether to trigger the hotspot index update condition includes: determining whether to trigger the hotspot index update condition based on at least one of file operation records, the storage status of a preset index structure, and hotspot file configuration adjustment information; if the file operation records indicate that a file has been changed or has not been operated on for a long time, or the storage status of the preset index structure is saturated, or the hotspot file configuration adjustment information is received, then it is determined that the hotspot index update condition is triggered; otherwise, it is determined that the hotspot index update condition is not triggered.

[0104] Continue to refer to Figure 4 Determining whether a hotspot index update condition has been triggered can be done from several aspects, including the user's file operations, the storage status of preset index results, and adjustments to the user's hotspot index configuration.

[0105] (1) Dynamic analysis of file operation records

[0106] By capturing file change operations (such as adding, modifying, and deleting) and whether there has been no operation for a long time, the system dynamically determines whether to trigger the conditions for updating hot indexes.

[0107] On the one hand, when files are added, modified, deleted, moved, or copied, it can be considered that the conditions for updating the hotspot index have been triggered. For example, after a user deletes the file "abc.pdf", the system detects the deletion event sent by the file system and removes all index information corresponding to the file "abc.pdf" from the preset index structure to avoid subsequent queries returning invalid results; if a user adds a hotspot file "123.docx", the system parses the file content and extracts keywords to build a hotspot index for it.

[0108] On the other hand, the access frequency of files can be periodically monitored. If the access frequency of a file is lower than a preset threshold over a period of time, it indicates that the user has not operated on the file for a long time, which can be considered as triggering the hotspot index update condition. For example, if the user's access frequency for the file "travel photos.jpg" is lower than the preset threshold, the file can be downgraded or cleaned up to update the index. For example, it can be downgraded to retain only its basic index information (filename, path) and delete deep index information (such as text content extracted by OCR), thereby saving storage space.

[0109] (2) Real-time monitoring of the storage status of the index structure

[0110] The system monitors storage status metrics of the preset index structure, such as memory usage and disk space utilization, and compares them with preset thresholds. If the index storage usage exceeds the threshold, a hot index update condition is triggered, which can downgrade or clean up the index data stored in the preset index structure. The principle of downgrading or cleaning up is to prioritize retaining frequently accessed index data and downgrade or clean up infrequently accessed index data, thereby avoiding index bloat and excessive resource consumption while maintaining retrieval efficiency.

[0111] In addition, you can set an upper limit for the index size and periodically check for infrequently accessed index records. When the index size limit is exceeded, the records can be compressed, downgraded, or cleaned up. When storage and computing resources are strained, you can also pause unnecessary updates to hot data indexes.

[0112] (3) Immediate response to user configuration adjustments

[0113] Users can proactively adjust the configuration information of hot files through the interface or API, such as adjusting the type and range of hot files, and trigger index updates in real time based on configuration changes.

[0114] Users can add specific file types (such as PDF and Excel) as new popular file types, or delete some existing popular file types. The system updates the index content based on the user's adjustments. For example, after a user adds PDF as a popular file type, the system performs a deep index on all PDF files (extracting the table of contents and keywords) and adds them to the preset index structure.

[0115] By employing a multi-dimensional index update triggering mechanism that tracks file operation records, index storage status, and user configuration adjustments, the system can accurately capture index information that needs updating, preventing missed or excessive updates. Differentiated index management strategies and adaptive scaling mechanisms significantly reduce index storage overhead, ensuring retrieval efficiency.

[0116] The above technical solution improves the accuracy and real-time performance of index updates while significantly reducing storage space usage. It can also adapt to diverse scenarios and user needs, providing users with efficient, stable, and low-resource-consumption file retrieval services.

[0117] In some embodiments of this application, it is supported to predict the types of files that a user may access using simple machine learning algorithms, and to build or update the index in advance based on the types of files that a user may access.

[0118] In summary, the key points and technical effects achieved in this application mainly include:

[0119] (1) Lightweight design reduces hardware dependence

[0120] This application fully utilizes the metadata (such as path, filename, size, modification time, etc.) provided by the file system, avoiding full indexing of all files and significantly reducing the demand on server-side hardware computing and storage resources. The hotspot index employs a hierarchical storage strategy, extracting necessary metadata only from frequently accessed files and storing it in a lightweight index structure, further reducing system resource consumption.

[0121] (2) Dynamic indexing mechanism improves resource utilization efficiency

[0122] This application designs a dynamic index management mechanism that can dynamically adjust the scope and index depth of hot files based on user access frequency and file operation records, thereby achieving efficient utilization of computing resources. The index content of infrequently accessed files can be periodically cleaned up or downgraded to ensure optimal system performance under limited resource conditions.

[0123] (3) Fast search, improving user experience

[0124] By storing basic information about frequently accessed files in a database and directly querying the hotspot index, the retrieval efficiency of these files is significantly improved. For ordinary files, the structured information of the file system is used directly for retrieval, eliminating the need for a full index and ensuring both high efficiency and low resource consumption in basic retrieval capabilities.

[0125] (4) Flexible adaptation and enhanced scalability

[0126] This application allows users to customize the type and scope of hot data according to their actual needs, such as indexing only specific directories or file types to meet personalized needs in different scenarios. By dynamically adjusting the indexing strategy, the system can adapt to the resource conditions of different devices (such as storage space and computing power), and still provide efficient management and retrieval capabilities on resource-constrained devices.

[0127] (5) Optimize the caching mechanism to reduce redundant calculations.

[0128] This application introduces a caching mechanism in basic file system information retrieval to avoid repeated file system read operations. Through a cache invalidation mechanism, the real-time nature and accuracy of retrieval results after file updates are ensured, while simultaneously reducing system computational overhead.

[0129] This application also provides a file management and retrieval device 500, applied to a storage service terminal, such as... Figure 5 The diagram shows a structural schematic of a file management and retrieval device according to an embodiment of this application. The file management and retrieval device 500 includes: a receiving unit 510, a first indexing unit 520, a first determining unit 530, a second indexing unit 540, and a retrieval unit 550, wherein:

[0130] The receiving unit 510 is used to receive files uploaded by the client;

[0131] The first indexing unit 520 is used to store the file in the file system of the storage service terminal and establish the basic index information of the file;

[0132] The first determining unit 530 is used to determine whether the file meets the first hotspot index condition;

[0133] The second indexing unit 540 is used to establish the deep index information of the file when the file meets the first hot spot indexing condition, and to store the basic index information of the file and the deep index information of the file into a preset index structure.

[0134] The retrieval unit 550 is used to perform file retrieval based on the index information of the file system or the index information of the preset index structure.

[0135] In some embodiments of this application, the first determining unit 530 is specifically used to: obtain hotspot file configuration information, the hotspot file configuration information including at least one of the types and ranges of hotspot files; and determine whether the file meets the first hotspot index condition based on the hotspot file configuration information.

[0136] In some embodiments of this application, the retrieval unit 550 is specifically used for: receiving a user's file query request; responding to the user's file query request, determining whether the file corresponding to the file query request can be found in the cache; if so, returning the file query result in the cache to the client; if not, performing a file retrieval in the index information of the file system or the index information of the preset index structure according to the file query request.

[0137] In some embodiments of this application, the retrieval unit 550 is specifically configured to: in response to the file query request, determine whether the file to be queried is a hot file; if so, query the index information of the preset index structure according to the file query request, obtain a first file query result and return it to the client, and store the first file query result in the cache; if not, query the index information of the file system according to the file query request, obtain a second file query result and return it to the client, and store the second file query result in the cache.

[0138] In some embodiments of this application, the file management and retrieval device 500 further includes: a second determining unit, configured to determine whether to trigger a cache invalidation mechanism based on file changes in the file system; and a cache invalidation unit, configured to determine invalidated data in the cache and clean it up if the cache invalidation mechanism is triggered.

[0139] In some embodiments of this application, the file management and retrieval device 500 further includes: a statistics unit, configured to count the queried data of a file based on the file query results in the cache; a third determination unit, configured to determine whether the file satisfies the second hot spot index condition based on the queried data of the file; and a second index unit, further configured to, when the file satisfies the second hot spot index condition, establish the deep index information of the file, and add the basic index information of the file and the deep index information of the file to the preset index structure.

[0140] In some embodiments of this application, the file management and retrieval device 500 further includes: a fourth determining unit, configured to determine whether a hotspot index update condition is triggered; and an index updating unit, configured to update the index information in the preset index structure when the hotspot index update condition is triggered.

[0141] In some embodiments of this application, the fourth determining unit is specifically used to: determine whether the hotspot index update condition is triggered based on at least one of file operation records, the storage status of a preset index structure, and hotspot file configuration adjustment information; if the file operation records determine that the file has been changed or has not been operated on for a long time, or the storage status of the preset index structure is saturated, or the hotspot file configuration adjustment information is received, then the hotspot index update condition is triggered; otherwise, the hotspot index update condition is not triggered.

[0142] It is understood that the above-mentioned file management and retrieval device can realize all the steps of the file management and retrieval method provided in the foregoing embodiments. The relevant explanations of the file management and retrieval method are applicable to the file management and retrieval device, and will not be repeated here.

[0143] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 6 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0144] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0145] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0146] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0147] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0148] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0149] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0150] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0151] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0152] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0153] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0154] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0155] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A file management and retrieval method, characterized in that, Applied to the storage service, the file management and retrieval method includes: Receive files uploaded by the client; The file is stored in the file system of the storage service client, and the basic index information of the file is obtained; Determine whether the file meets the first hotspot indexing condition; If the file meets the first hotspot indexing condition, a deep index information for the file is established, and the basic index information and the deep index information of the file are stored in a preset index structure. File retrieval is performed based on the index information of the file system or the index information of the preset index structure.

2. The file management and retrieval method according to claim 1, characterized in that, Determining whether the file meets the first hotspot indexing condition includes: Obtain hotspot file configuration information, wherein the hotspot file configuration information includes at least one of the types and ranges of hotspot files; Based on the hotspot file configuration information, determine whether the file meets the first hotspot index condition.

3. The file management and retrieval method according to claim 1, characterized in that, The process of retrieving files based on the index information of the file system or the index information of the preset index structure includes: Receive file query requests from users; In response to a user's file query request, determine whether the file corresponding to the file query request can be found in the cache; If possible, the cached file query results will be returned to the client. If not, then file retrieval is performed in the file system's index information or the index information of the preset index structure according to the file query request.

4. The file management and retrieval method according to claim 3, characterized in that, The step of performing file retrieval based on the file query request in the index information of the file system or the index information of the preset index structure includes: In response to the file query request, determine whether the file to be queried is a hot file; If so, then according to the file query request, a query is performed in the index information of the preset index structure to obtain the first file query result and return it to the client, and the first file query result is stored in the cache; If not, then according to the file query request, a query is performed in the index information of the file system to obtain the second file query result and return it to the client, and the second file query result is stored in the cache.

5. The file management and retrieval method according to claim 3 or 4, characterized in that, The document management and retrieval method also includes: Determine whether to trigger the cache invalidation mechanism based on file changes in the file system; If the cache invalidation mechanism is triggered, invalid data in the cache is identified and cleaned up.

6. The file management and retrieval method according to claim 3 or 4, characterized in that, The document management and retrieval method also includes: The query data of the file is statistically analyzed based on the file query results in the cache. Determine whether the file meets the second hotspot indexing condition based on the queried data of the file; If the file satisfies the second hotspot indexing condition, a deep index information for the file is established, and the basic index information and the deep index information of the file are added to the preset index structure.

7. The file management and retrieval method according to claim 1, characterized in that, The document management and retrieval method also includes: Determine whether the hotspot index update condition has been triggered; When the hotspot index update condition is triggered, the index information in the preset index structure is updated.

8. The file management and retrieval method according to claim 7, characterized in that, The conditions for determining whether to trigger a hotspot index update include: Whether the hotspot index update condition is triggered is determined based on at least one of the following: file operation records, storage status of the preset index structure, and hotspot file configuration adjustment information. If, based on the file operation records, it is determined that a file has been modified or has not been operated on for a long time, or the storage status of the preset index structure is saturated, or the hot file configuration adjustment information is received, then the conditions for triggering the hot index update are determined. Otherwise, it is determined that the hotspot index update condition has not been triggered.

9. A document management and retrieval device, characterized in that, The file management and retrieval device, used in storage services, includes: The receiving unit is used to receive files uploaded by the client; The first indexing unit is used to store the file in the file system of the storage service terminal and establish the basic index information of the file; The first determining unit is used to determine whether the file satisfies the first hotspot index condition; The second indexing unit is used to establish the deep index information of the file when the file meets the first hot spot indexing condition, and to store the basic index information and the deep index information of the file into a preset index structure. The retrieval unit is used to perform file retrieval based on the index information of the file system or the index information of the preset index structure.

10. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the file management and retrieval methods of claims 1 to 8.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the file management and retrieval method according to any one of claims 1 to 8.

Images