Distributed system file management method and device based on neural network and medium

Abstract

The invention discloses a distributed system file management method and device based on a neural network and a medium, and the method comprises the steps: extracting small file features through a convolutional neural network to construct a file index, and analyzing a large file through a recurrent neural network to segment the large file into a plurality of sub-blocks; analyzing an access log of the segmented sub-blocks through a long-short-term memory network to predict an access mode, distributing the file and the sub-blocks to corresponding levels in a four-level storage layer according to the access mode, and monitoring the access mode change of the file and the sub-blocks to trigger cross-layer dynamic migration; before data transmission, a to-be-transmitted file is preprocessed through a source client, a data packet and a processing rule are packaged as a transmission unit, and a transmission link state is monitored in real time through a recurrent neural network; and dynamically selecting a transmission link to send the transmission unit to a receiving end, analyzing the transmission unit through the receiving end, storing the data packet to a corresponding level in the four-level storage layer based on the analyzed processing rule information, and performing data integrity verification.

Description

Technical Field

[0001] This application relates to the field of file management technology, and in particular to a method, device and medium for file management in a distributed system based on neural networks. Background Technology

[0002] In the field of distributed systems technology, facing the ever-increasing demand for storage and transmission of massive amounts of small files, especially in resource-constrained application environments for small and medium-sized enterprises, existing technical solutions have significant shortcomings in terms of efficiency, cost, and intelligence. Existing storage architectures mostly employ fixed or coarse-grained tiering strategies, lacking fine-grained perception and dynamic response capabilities regarding file access characteristics, making it difficult to achieve an optimal balance between performance and cost under limited hardware resources. Storage resource allocation is often based on static strategies or simple thresholds, failing to adapt to dynamic changes in access patterns. This results in frequently accessed files not being placed on high-speed storage media, while infrequently accessed files occupy valuable high-speed resources, leading to low resource utilization and business response delays.

[0003] Furthermore, traditional file processing methods have significant shortcomings in redundancy elimination and content management. Deduplication techniques based on hash algorithms cannot identify files with similar but not identical content, leaving behind substantial storage redundancy. Mechanically splitting large files into fixed-size segments disrupts their inherent logical coherence, reducing the efficiency of subsequent processing. At the data transmission level, existing solutions lack unified encapsulation and synchronous transmission of data packets and processing rules, requiring additional parsing and processing at the receiving end, increasing bandwidth consumption and raising the technical requirements for maintenance personnel.

[0004] Furthermore, existing systems generally lack AI-based predictive and decision-making capabilities. Determining file popularity relies heavily on historical statistics, failing to accurately predict future access trends and causing delays in storage migration and resource scheduling strategies. Simultaneously, the lack of real-time network status awareness and intelligent adaptation during transmission prevents dynamic selection of optimal transmission paths for files with varying requirements, a problem particularly pronounced in bandwidth-constrained network environments. Regarding data security, data in volatile memory lacks efficient and reliable real-time backup mechanisms, posing a high risk of data loss. Summary of the Invention

[0005] This application provides a method, device, and medium for managing distributed system files based on neural networks to solve the aforementioned technical problems.

[0006] On one hand, embodiments of this application provide a distributed system file management method based on neural networks, including: A convolutional neural network is used to extract features from small files in the file to be managed, and a file index is constructed. A recurrent neural network is used to analyze large files in the file to be managed and divide them into several sub-blocks. The access logs of the segmented sub-blocks are analyzed by Long Short-Term Memory Network to predict access patterns. Based on the access patterns, the files and sub-blocks are allocated to the corresponding levels in the four-level storage layer. The changes in the access patterns of the files and sub-blocks are monitored to trigger dynamic cross-level migration. Before data transmission, the source client preprocesses the file to be transmitted, encapsulates data packets and processing rules into transmission units, and monitors the transmission link status in real time through a recurrent neural network. The transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit, stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification.

[0007] In one implementation of this application, a convolutional neural network is used to extract features from small files in the file to be managed and construct a file index, specifically including: Multidimensional feature vectors of small files in the files to be managed are extracted using a convolutional neural network, and the cosine similarity between the multidimensional feature vectors is calculated. If the cosine similarity exceeds a preset similarity threshold, it is determined that there are duplicate files in the small file, so that the small file is deduplicated and an index is built on the deduplicated file.

[0008] In one implementation of this application, a large file in the file to be managed is analyzed using a recurrent neural network and divided into several sub-blocks, specifically including: The semantic relationships of the file content of large files in the files to be managed are analyzed by a recurrent neural network, and the segmentation threshold is dynamically adjusted according to the semantic relationships to ensure that the segmented sub-blocks maintain logical integrity. The file is divided into several sub-blocks according to the segmentation threshold, and an independent access log is generated for each sub-block.

[0009] In one implementation of this application, access logs of the segmented sub-blocks are analyzed using a Long Short-Term Memory (LSTM) network to predict access patterns. Based on these access patterns, files and sub-blocks are allocated to corresponding levels in a four-level storage hierarchy. Specifically, this includes: Collect access logs for the file and each segmented sub-block; the access logs include access timestamps, frequency, and latency requirements; The access logs are analyzed using a long short-term memory network to predict access trends in future time periods based on historical access sequences. Based on the prediction results, files and sub-blocks are allocated to the corresponding levels in the four-level storage tier consisting of memory, solid-state drives, hard disk drives, and tape libraries.

[0010] In one implementation of this application, changes in the access patterns of files and sub-blocks are monitored to trigger dynamic cross-layer migration, specifically including: If the access frequency of a file stored in memory drops below a first frequency threshold, the file will be migrated to a solid-state drive. If the access frequency of a file stored on the solid-state drive drops below a second frequency threshold, the file will be migrated to the hard disk drive. If a file stored on a mechanical hard drive remains unaccessed for a period exceeding a preset time threshold, the file is marked as cold data, and the file marked as cold data is migrated to a tape library.

[0011] In one implementation of this application, it further includes: The system monitors file modification operations in memory and, upon detecting that a file in memory has been modified, triggers a synchronized backup of the file to the solid-state drive. Generate a verification code and compare the backup data with the memory data to see if they match. For files whose modification frequency exceeds a preset ultra-high frequency threshold, a combination of batch asynchronous backup and real-time log recording is used for processing.

[0012] In one implementation of this application, before data transmission, the source client preprocesses the file to be transmitted, encapsulating data packets and processing rules into transmission units, specifically including: Before data transmission, the source client performs deduplication, segmentation, and access mode marking on the file to be transmitted; The data packet, feature digest, processing rules, and checksum are encapsulated into a unified transmission unit; the processing rules include deduplication threshold, segmentation logic, and storage layer allocation parameters.

[0013] In one implementation of this application, a transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit, stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification. Specifically, this includes: The gated loop unit is used to predict the latency of each available transmission link and identify transmission links whose predicted latency is lower than a predetermined low latency threshold. Files or sub-blocks marked as UHF access and requiring low latency will be assigned to transmission links with a predicted latency below a predetermined low latency threshold for transmission. The access mode tag rule parameters in the processing rule information are parsed, and the target storage layer corresponding to the data packet is determined according to the access mode tag rule parameters, so as to store the data packet in the determined target storage layer.

[0014] On the other hand, embodiments of this application also provide a distributed system file management device based on neural networks, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are then executed by the at least one processor to enable the at least one processor to perform the neural network-based distributed system file management method described above.

[0015] On the other hand, embodiments of this application also provide a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned distributed system file management method based on neural networks.

[0016] This application provides a distributed system file management method, device, and medium based on neural networks, which have at least the following beneficial effects: By leveraging convolutional neural networks for deep feature extraction and similarity comparison, accurate deduplication based on file content features is achieved, reducing storage redundancy. Recurrent neural networks analyze the semantic and logical relationships of file content and dynamically segment it, ensuring the logical integrity and semantic coherence of the segmented sub-blocks. Long Short-Term Memory (LSTM) networks analyze historical access logs to predict future access patterns, and accordingly, files and sub-blocks are accurately allocated to a four-tier storage system, achieving optimal matching of storage resources and file access frequency, improving the utilization efficiency of high-speed storage resources and reducing overall storage costs. Continuous monitoring of access pattern changes and triggering dynamic cross-tier migration enable adaptive storage layout. Despite evolving business needs, we maintain highly efficient resource allocation. By preprocessing data and processing rules on the source client side and encapsulating them into a unified transmission unit, the receiving end can directly store data according to the rules without complex secondary parsing, significantly improving transmission and storage efficiency. Recurrent neural networks monitor the real-time status of the transmission link and dynamically select the optimal link for files with different requirements, effectively ensuring low-latency transmission for high-frequency file access and optimizing the utilization of limited network bandwidth. By embedding checksums and integrity verification steps, we ensure the consistency and reliability of data during transmission and storage, greatly reducing the risk of data loss and errors. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a distributed system file management method based on neural networks provided in an embodiment of this application; Figure 2 This is a schematic diagram of the internal structure of a distributed system file management device based on a neural network, provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a distributed system file management method based on neural networks, provided in an embodiment of this application.

[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0022] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0023] like Figure 1 As shown in the embodiments of this application, the distributed system file management method based on neural networks includes: Step 101: Use a convolutional neural network to extract features from small files in the file to be managed, build a file index, and use a recurrent neural network to analyze large files in the file to be managed and divide them into several sub-blocks.

[0024] In this embodiment, for small files within the files to be managed, the system invokes a pre-trained Convolutional Neural Network (CNN) model for feature extraction. Through its unique convolutional and pooling layer structure, the CNN can learn and extract high-dimensional, semantically meaningful deep features from the file's binary stream or parsed content, outputting a fixed-length feature vector. Specifically, this feature vector represents the core content of the file, rather than its surface form. For example, for text files, the CNN can extract semantic features such as paragraph structure and keyword distribution; for image files, it can extract visual information such as color distribution and texture features. It should be noted that these feature extraction processes do not rely on simple file hash values, but rather obtain more discriminative feature representations through multi-layered nonlinear transformations of the neural network.

[0025] Specifically, after feature extraction, the system calculates the cosine similarity between the feature vector of the file to be processed and the feature vectors of already stored files. For example, if the calculated maximum cosine similarity exceeds a preset similarity threshold, the new file is determined to be a duplicate of the corresponding file in the database. The system will discard the new file, retaining only one copy of the data, thus achieving accurate content-based deduplication. It is understandable that this content-based deduplication method is better at identifying files with similar content but different formats compared to traditional hash comparisons. The deduplicated files are assigned a unique identifier and indexed, with index information including feature vectors, storage locations, and other metadata.

[0026] In this embodiment, large files exceeding a preset size threshold are processed using a Recurrent Neural Network (RNN). Through its recurrent connection structure, the RNN effectively analyzes the temporal correlation of file content. It should be noted that the RNN adaptively adjusts its segmentation strategy for different file types. Specifically, for a long text report, the RNN can identify the boundaries between chapters and sub-chaps; for a structured database export file, the RNN can perceive the grouping patterns between records. Based on a deep analysis of the logical features of this content, the RNN model dynamically generates one or more segmentation thresholds. These thresholds are not fixed in size but are determined based on the logical units of the content. Finally, the system segments the large file into several semantically coherent and complete sub-blocks according to these dynamic thresholds. Each sub-block is assigned a unique identifier and generates an independent access log, laying the foundation for subsequent refined and differentiated management and storage.

[0027] Step 102: Analyze the access logs of the segmented sub-blocks using a long short-term memory network to predict access patterns. Based on the access patterns, allocate the files and sub-blocks to the corresponding levels in the four-level storage layer and monitor changes in the access patterns of the files and sub-blocks to trigger dynamic cross-level migration.

[0028] In this embodiment, the access log records detailed metadata for each access, with core parameters including the timestamp of the access, the frequency of the access, and the implicit or explicit latency requirements of the access operation. The system utilizes a Long Short-Term Memory (LSTM) network model for in-depth analysis of the time-series log data. The LSTM model, through its unique gating mechanism, can effectively capture long-term dependencies in the access log. For example, the system collects access logs for files and sub-blocks, including key indicators such as access timestamps and frequencies. It should be noted that this log data undergoes standardization to eliminate biases caused by different time scales.

[0029] The LSTM network analyzes historical access sequences to predict file access trends in the future. Based on the predictions, the system allocates files to four storage tiers. Specifically, the memory tier stores frequently accessed files, the solid-state drive tier stores frequently accessed files, the hard disk drive tier stores infrequently accessed files, and the tape library tier stores cold data. This tiered storage strategy achieves an optimal balance between performance and cost, placing the most frequently accessed data on the fastest media and the least frequently accessed data on the cheapest media.

[0030] In this embodiment, the file access pattern is dynamically changing, so the system continuously monitors it. When a significant change in the access pattern of an entity is detected, a cross-layer dynamic migration is triggered. Specifically, when it is found that the access frequency of a file stored in memory has been continuously decreasing and has fallen below a frequency threshold set for high-frequency access, the file is migrated to the NVMe SSD layer; when it is found that the access frequency of a file in the NVMe SSD has further decreased to a low-frequency threshold, the file is migrated to the SATA mechanical hard drive layer; and for files in the SATA mechanical hard drive that have not been accessed for more than a preset period of time, the files are marked as cold data and migrated to the tape library for archiving storage.

[0031] It should be noted that, to ensure data security, this application also includes a memory backup mechanism. The system monitors modification operations on files in memory and triggers a synchronous backup of the file to the NVMe SSD when a modification occurs. After the backup is complete, the system generates checksums for both the source and backup files and compares them to ensure the integrity and consistency of the backup. For files with extremely high modification frequency, a strategy combining batch asynchronous backup with real-time modification logging can be used to balance performance overhead.

[0032] Step 103: Before data transmission, the source client preprocesses the file to be transmitted, encapsulates data packets and processing rules into transmission units, and monitors the transmission link status in real time through a recurrent neural network.

[0033] In this embodiment, source-side preprocessing includes operations such as deduplication, segmentation, and access pattern marking. It is understood that these preprocessing operations can significantly reduce the amount of data transmitted and improve transmission efficiency. For example, the system performs feature extraction and similarity comparison on the files to be transmitted to remove duplicate content; large files are intelligently segmented to maintain the logical integrity of the content.

[0034] Specifically, after preprocessing, the system encapsulates data packets, feature summaries, processing rules, and other information into a unified transmission unit. It's important to note that this encapsulation method enables synchronous transmission of data and rules, allowing the receiving end to directly execute storage operations according to the rules. Understandably, the processing rules include key parameters such as deduplication thresholds and segmentation logic to ensure consistency throughout the processing.

[0035] Meanwhile, during transmission, the system uses a recurrent neural network (RNN), such as a gated recurrent unit (GRU), to monitor the status of all available transmission links in real time. GRU networks are well-suited for processing time-series data, as they can predict the short-term performance of links based on their historical status data, such as latency, throughput, and packet loss rate, especially predicting link latency.

[0036] Step 104: Dynamically select the transmission link according to the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit, stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification.

[0037] In this embodiment, the system dynamically selects the most suitable link for the transmission unit waiting to be transmitted based on the link latency predictions of the GRU network. For example, for files or sub-blocks marked as having ultra-high frequency access and requiring low latency during the preprocessing stage, the system will prioritize allocating the high-quality link with the lowest currently predicted latency to ensure its transmission speed and meet the stringent low latency requirements of the service.

[0038] After the transmission unit arrives at the receiving end, the receiving end does not need to perform complex rule inference or content analysis. Understandably, the receiving end can directly parse the structured information of the transmission unit to separate the data packet, processing rules, and checksum. The receiving end then automatically stores the data packet in the corresponding target level of its local four-level storage system based on the access mode markers and storage layer allocation parameters explicitly indicated in the parsed processing rules. This greatly simplifies the receiving end's processing logic and reduces the computational requirements on the receiving end.

[0039] Finally, to ensure that data has not been tampered with or erroneous during transmission, the receiving end performs a data integrity check. It's important to note that the receiving end uses the same checksum algorithm as the source end to calculate a checksum on the received data packet, then compares this value with the source checksum carried in the transmission unit. If they match perfectly, the data transmission is considered complete and error-free; if they don't match, a retransmission mechanism is automatically triggered, and the receiving end requests a retransmission of the transmission unit from the source end, thus ensuring end-to-end data reliability. Understandably, this end-to-end checksum mechanism significantly reduces the risk of data transmission errors.

[0040] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a distributed system file management device based on a neural network, the structure of which is as follows: Figure 2 As shown.

[0041] Figure 2 This is a schematic diagram of the internal structure of a distributed system file management device based on a neural network, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: A convolutional neural network is used to extract features from small files in the file to be managed, and a file index is constructed. A recurrent neural network is used to analyze large files in the file to be managed and divide them into several sub-blocks. The access logs of the segmented sub-blocks are analyzed by Long Short-Term Memory Network to predict access patterns. Based on the access patterns, files and sub-blocks are allocated to the corresponding levels in the four-level storage layer. Changes in the access patterns of files and sub-blocks are monitored to trigger dynamic cross-level migration. Before data transmission, the source client preprocesses the file to be transmitted, encapsulates data packets and processing rules into transmission units, and monitors the transmission link status in real time through a recurrent neural network. The transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit and stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification.

[0042] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: A convolutional neural network is used to extract features from small files in the file to be managed, and a file index is constructed. A recurrent neural network is used to analyze large files in the file to be managed and divide them into several sub-blocks. The access logs of the segmented sub-blocks are analyzed by Long Short-Term Memory Network to predict access patterns. Based on the access patterns, files and sub-blocks are allocated to the corresponding levels in the four-level storage layer. Changes in the access patterns of files and sub-blocks are monitored to trigger dynamic cross-level migration. Before data transmission, the source client preprocesses the file to be transmitted, encapsulates data packets and processing rules into transmission units, and monitors the transmission link status in real time through a recurrent neural network. The transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit and stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification.

[0043] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0044] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0045] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0049] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0050] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0051] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0052] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0053] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A distributed system file management method based on neural networks, characterized in that, The method includes: A convolutional neural network is used to extract features from small files in the file to be managed, and a file index is constructed. A recurrent neural network is used to analyze large files in the file to be managed and divide them into several sub-blocks. The access logs of the segmented sub-blocks are analyzed by Long Short-Term Memory Network to predict access patterns. Based on the access patterns, the files and sub-blocks are allocated to the corresponding levels in the four-level storage layer. The changes in the access patterns of the files and sub-blocks are monitored to trigger dynamic cross-level migration. Before data transmission, the source client preprocesses the file to be transmitted, encapsulates data packets and processing rules into transmission units, and monitors the transmission link status in real time through a recurrent neural network. The transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit, stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification.

2. The distributed system file management method based on neural networks according to claim 1, characterized in that, A convolutional neural network is used to extract features from small files in the files to be managed and to build a file index, specifically including: Multidimensional feature vectors of small files in the files to be managed are extracted using a convolutional neural network, and the cosine similarity between the multidimensional feature vectors is calculated. If the cosine similarity exceeds a preset similarity threshold, it is determined that there are duplicate files in the small file, so that the small file is deduplicated and an index is built on the deduplicated file.

3. The distributed system file management method based on neural networks according to claim 1, characterized in that, The large files in the files to be managed are analyzed using a recurrent neural network and divided into several sub-blocks, specifically including: The semantic relationships of the file content of large files in the files to be managed are analyzed by a recurrent neural network, and the segmentation threshold is dynamically adjusted according to the semantic relationships to ensure that the segmented sub-blocks maintain logical integrity. The file is divided into several sub-blocks according to the segmentation threshold, and an independent access log is generated for each sub-block.

4. The distributed system file management method based on neural networks according to claim 1, characterized in that, Access patterns are predicted by analyzing the access logs of the segmented sub-blocks using a Long Short-Term Memory (LSTM) network. Based on these access patterns, files and sub-blocks are then allocated to corresponding levels within a four-level storage hierarchy. Specifically, this includes: Collect access logs for the file and each segmented sub-block; the access logs include access timestamps, frequency, and latency requirements; The access logs are analyzed using a long short-term memory network to predict access trends in future time periods based on historical access sequences. Based on the prediction results, files and sub-blocks are allocated to the corresponding levels in the four-level storage tier consisting of memory, solid-state drives, hard disk drives, and tape libraries.

5. The distributed system file management method based on neural networks according to claim 1, characterized in that, Monitor changes in access patterns of files and sub-blocks to trigger dynamic cross-layer migrations, specifically including: If the access frequency of a file stored in memory drops below a first frequency threshold, the file will be migrated to a solid-state drive. If the access frequency of a file stored on the solid-state drive drops below a second frequency threshold, the file will be migrated to the hard disk drive. If a file stored on a mechanical hard drive remains unaccessed for a period exceeding a preset time threshold, the file is marked as cold data, and the file marked as cold data is migrated to a tape library.

6. The distributed system file management method based on neural networks according to claim 5, characterized in that, The method further includes: The system monitors file modification operations in memory and, upon detecting that a file in memory has been modified, triggers a synchronized backup of the file to the solid-state drive. Generate a verification code and compare the backup data with the memory data to see if they match. For files whose modification frequency exceeds a preset ultra-high frequency threshold, a combination of batch asynchronous backup and real-time log recording is used for processing.

7. The distributed system file management method based on neural networks according to claim 1, characterized in that, Before data transmission, the source client preprocesses the file to be transmitted, encapsulating data packets and processing rules into transmission units, specifically including: Before data transmission, the source client performs deduplication, segmentation, and access mode marking on the file to be transmitted; The data packet, feature digest, processing rules, and checksum are encapsulated into a unified transmission unit; the processing rules include deduplication threshold, segmentation logic, and storage layer allocation parameters.

8. The distributed system file management method based on neural networks according to claim 1, characterized in that, The transmission link is dynamically selected based on the transmission link status to send the transmission unit to the receiving end. The receiving end parses the transmission unit, stores the data packet in the corresponding level of the four-level storage layer based on the parsed processing rule information, and performs data integrity verification, specifically including: The gated loop unit is used to predict the latency of each available transmission link and identify transmission links whose predicted latency is lower than a predetermined low latency threshold. Files or sub-blocks marked as UHF access and requiring low latency will be assigned to transmission links with a predicted latency below a predetermined low latency threshold for transmission. The access mode tag rule parameters in the processing rule information are parsed, and the target storage layer corresponding to the data packet is determined according to the access mode tag rule parameters, so as to store the data packet in the determined target storage layer.

9. A distributed system file management device based on neural networks, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the neural network-based distributed system file management method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement the distributed system file management method based on neural networks as described in any one of claims 1-8.

Images