File reading method and apparatus

Abstract

The application discloses a file reading method and device. The method comprises the following steps: in response to an access request from a target device for accessing a target file, searching for the target file from a target cache, wherein a first quantity of first files are pre-stored in the target cache, the first files are files with high target scores in a plurality of second files in a target data storage system, and the target score is a score for reflecting whether the second file is pre-stored, which is determined based on an analytic hierarchy process; when the target cache comprises the target file, feeding back the target file to the target device; when the target cache does not comprise the target file, reading the target file from the target data storage system, storing the read target file into the target cache, and feeding back the target file to the target device. The application solves the technical problem that the data storage system is prone to freezing and downtime in a high-concurrency file access scenario in the related art.

Description

Technical Field

[0001] This application relates to the field of software development technology, and more specifically, to a file reading method and apparatus. Background Technology

[0002] Currently, with the continuous advancement of digital transformation, digital systems are being developed and launched at the unit and town levels. However, due to frequent sudden events in these digital systems, scenarios arise involving high-concurrency access to large batches of frequently accessed documents, event posters, and other files. Related technologies typically use open-source caching components to store frequently accessed small files such as promotional documents and posters, then pre-cache them using a basic LRU (Least Recently Used) eviction algorithm, updating the frequently accessed small files and saving their characteristic keys. For file keys that are not found, small files are read and accessed through a multi-level directory file system.

[0003] However, this method often has the following drawbacks in production activities: First, if the open-source caching component used is a single-threaded service process, it will result in low CPU utilization. Although it supports automatic recovery in terms of single-point failure recovery, the process is prone to data corruption. Second, if the system cache is full or the system faces cache breakdown, the multi-directory file system will access the hard drive more often when querying files due to its own characteristics, resulting in performance loss, reduced file query efficiency, and problems such as lag and crashes are likely to occur in high-concurrency access scenarios.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a file reading method and apparatus to at least solve the technical problems of data storage systems in the related art being prone to lag and crashing in high-concurrency file access scenarios.

[0006] According to one aspect of the embodiments of this application, a file reading method is provided, comprising: responding to an access request from a target device for accessing a target file, searching for a target file in a target cache, wherein the target cache pre-caches a first number of first files, the first files being the files with the highest target scores among a plurality of second files in a target data storage system, and the target score being a score determined based on the analytic hierarchy process (AHP) to reflect whether the second files are pre-cached; when the target cache includes a target file, sending the target file back to the target device; when the target cache does not include a target file, reading the target file from the target data storage system, storing the read target file in the target cache, and sending the target file back to the target device.

[0007] Optionally, before searching for the target file in the target cache, the method further includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: business importance of the file, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, or not pre-caching the second file; calculating a target score for each second file based on the first weight result and the second weight result; sorting multiple second files from largest to smallest based on the target score, and taking the first number of second files at the top of the sort as the first file, and pre-caching the first file to the target cache.

[0008] Optionally, before calculating the first weight result of the criterion-level judgment matrix, the method further includes: performing a consistency check on the criterion-level judgment matrix; calculating the first weight result of the criterion-level judgment matrix when the criterion-level judgment matrix passes the consistency check; and readjusting the values ​​of the criterion-level judgment matrix when the criterion-level judgment matrix fails the consistency check. Before calculating the second weight result of the scheme-level judgment matrix, the method further includes: performing a consistency check on the scheme-level judgment matrix; calculating the second weight result of the scheme-level judgment matrix when the scheme-level judgment matrix passes the consistency check; and readjusting the values ​​of the scheme-level judgment matrix when the scheme-level judgment matrix fails the consistency check.

[0009] Optionally, calculating the first weight result of the criterion-level judgment matrix includes: determining the first weight of each indicator in the criterion-level judgment matrix using the geometric mean method to obtain the first weight result; calculating the second weight result of the scheme-level judgment matrix includes: determining the second weight of each scheme in the scheme-level judgment matrix using the geometric mean method to obtain the second weight result.

[0010] Optionally, after storing the target file in the target cache, the method further includes: deleting the first file with the lowest target score from the target cache.

[0011] Optionally, the multiple second files in the target data storage system are stored based on a hash structure; the third file among the multiple second files is stored in the first type of resource pool in the target data storage system, and the fourth file among the multiple second files is stored in the second type of resource pool in the target data storage system. The third file is a file among the multiple second files whose target score is higher than a preset threshold, and the fourth file is a file among the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than that of the second type of resource pool.

[0012] Optionally, the target cache is a distributed caching module based on ctg-cache.

[0013] According to another aspect of the embodiments of this application, a file reading device is also provided, comprising: a search module, configured to search for a target file from a target cache in response to an access request from a target device for accessing a target file, wherein the target cache pre-caches a first number of first files, the first files being the files ranked first in a target score among a plurality of second files in a target data storage system, and the target score being a score determined based on the analytic hierarchy process (AHP) to reflect whether the second files are pre-cached; a first feedback module, configured to directly feed the target file back to the target device when the target cache includes the target file; and a second feedback module, configured to read the target file from the target data storage system, store the read target file in the target cache, and feed the target file back to the target device when the target cache does not include the target file.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein the device where the non-volatile storage medium is located executes the above-described file reading method by running the program.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described file reading method through the computer program.

[0016] In this embodiment, in response to an access request from the target device for accessing a target file, the target file is searched in the target cache. The target cache pre-caches a first number of first files, which are the files ranked highest in the target score among multiple second files in the target data storage system. The target score is a score determined based on the analytic hierarchy process (AHP) to reflect whether a second file should be pre-cached. If the target file is included in the target cache, it is sent back to the target device. If the target file is not included in the target cache, it is read from the target data storage system, stored in the target cache, and then sent back to the target device. The AHP-based target score ensures that the multiple second files in the target data storage system better meet business and performance requirements. This effectively improves the efficiency and stability of the target data storage system in reading files, thereby solving the technical problem of data storage systems easily experiencing lag and crashes under high-concurrency file access scenarios in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart illustrating a file reading method according to an embodiment of this application;

[0019] Figure 2 This is a hierarchical structure diagram of an optional analytic hierarchy process according to an embodiment of this application;

[0020] Figure 3 This is an illustration of an optional distributed caching module based on ctg-cache according to an embodiment of this application;

[0021] Figure 4a This is a schematic diagram of an optional data system directory tree structure according to an embodiment of this application;

[0022] Figure 4b This is a schematic diagram of an optional hash storage structure according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of an optional storage resource pool structure according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of a file reading device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0028] The Analytic Hierarchy Process (AHP) is a comprehensive evaluation method, essentially a process of decomposing complex problems. Through three basic steps—decomposition, judgment, and synthesis—this method reduces, to some extent, the influence of decision-maker subjectivity on the outcome. When executed, AHP generally involves four steps: First, analyzing the relationships between factors in the system to establish a hierarchical structure; next, comparing elements within the same level pairwise to construct a comparison matrix; then, calculating the relative weights of elements in each level's comparison matrix and performing a consistency check on the comparison matrices; finally, calculating the composite weights for the entire system and the total score of the evaluated object. The hierarchical structure refers to AHP's decomposition of the decision into an objective layer, a criterion layer, and a solution layer.

[0029] Hash storage refers to establishing a definite correspondence between the storage location of a record and its key, so that each key corresponds to a storage location. The location where an element is stored is called a "bucket," and each bucket has its own index, allowing the system to quickly access the elements stored within that bucket.

[0030] Example 1

[0031] Currently, with the continuous advancement of digital transformation, digital systems are being developed and launched at the unit and town levels. However, due to frequent sudden events in these digital systems, scenarios arise involving high-concurrency access to large batches of frequently accessed documents, event posters, and other files. Related technologies typically use open-source caching components to store these frequently accessed small files, such as promotional documents and posters. A basic LRU eviction algorithm is then used to pre-cache these small files, updating them in real time and saving their characteristic keys. For features of small files that are not matched, they can be read and accessed through a multi-level directory file system.

[0032] However, this method often has the following drawbacks in production activities: First, when considering factors such as business importance and user experience, using the basic LRU file pre-caching eviction algorithm may lead to distortion of cached results due to early eviction; Second, if the open-source caching component used is a single-threaded service process, it will result in low CPU utilization, and although automatic recovery is supported for single-point failures, the process is prone to data corruption; Third, if the system cache is full or the system faces cache breakdown, the multi-directory file system, due to its inherent characteristics, will access the hard drive more frequently when querying files, causing performance loss, reducing file query efficiency, and easily leading to stuttering and crashes in high-concurrency access scenarios; Fourth, using the same backup level storage resource pool for different files will result in a waste of storage resources.

[0033] To address the aforementioned problems, this application provides a file reading method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 1 This is a flowchart illustrating an optional file reading method according to an embodiment of this application, such as... Figure 1 As shown, the method includes at least steps S102-S106, wherein:

[0035] Step S102: In response to an access request from the target device for accessing a target file, the target file is searched in the target cache. The target cache contains a first number of first files, which are the files with the highest target scores among multiple second files in the target data storage system. The target score is a score determined based on the analytic hierarchy process to reflect whether the second files are pre-cached.

[0036] According to an optional embodiment of this application, before searching for the target file in the target cache, the target cache can be determined through the following steps S1021-S1024, wherein:

[0037] S1021, Establish the criterion layer judgment matrix and calculate the first weight result of the criterion layer judgment matrix. The criterion layer judgment matrix shall include at least the following indicators: business importance of the file, user experience, recent access time and file size.

[0038] Specifically, Figure 2This diagram illustrates the hierarchical structure of an optional analytic hierarchy process (AHP). The target layer selects objects to be evicted, i.e., unnecessary cached objects. The metric layer defines the eviction criteria, which are divided into four metrics: "file's business importance, user experience, recent access time, and file size." The solution layer determines whether to retain or evict a file based on current business needs. Figure 2 The hierarchical structure shown makes file pre-storage and cache eviction results more in line with business and performance requirements.

[0039] The criterion layer judgment matrix is ​​determined based on four indicators: "business importance of the file, user experience, recent access time and file size", as shown in Table 1.

[0040]

[0041]

[0042] The meaning of each scale in Table 1 can be found in the scale table, as shown in Table 2.

[0043] Table 2

[0044] Scale meaning 1 Equally important 3 Slightly important 5 Obviously important 7 Strongly important 9 Extremely important 2，4，6，8 The median of two adjacent values reciprocal If the scale of A compared to B is 3, then the scale of B compared to A is 1 / 3.

[0045] Optionally, before calculating the first weight result of the criterion layer judgment matrix, it is also necessary to perform a consistency check on the criterion layer judgment matrix A. If the criterion layer judgment matrix passes the consistency check, the first weight result of the criterion layer judgment matrix is ​​directly calculated; if the criterion layer judgment matrix fails the consistency check, the value of the criterion layer judgment matrix is ​​readjusted.

[0046] Typically, before performing consistency checks on the criterion-level judgment matrix, it is necessary to first determine whether the criterion-level judgment matrix satisfies all the conditions for a positive reciprocal matrix. Therefore, Table 1 can be written as the criterion-level judgment matrix, and the expression for the criterion-level judgment matrix is ​​as follows:

[0047]

[0048] It is easy to see that the criterion-level judgment matrix A is a square matrix; each data A in the criterion-level judgment matrix A... ij Each represents the relative importance of indicator j compared to indicator i. When i = j, it indicates that the two indicators are equally important. Therefore, the diagonal elements of the criterion layer judgment matrix A are all 1; each element in the criterion layer judgment matrix A is greater than zero, and A ij =A ji .

[0049] As can be seen from the above judgment conditions, the criterion layer judgment matrix A satisfies all the conditions of a positive reciprocal matrix, thus determining that the criterion layer judgment matrix A is a positive reciprocal matrix.

[0050] Furthermore, since the criterion-level judgment matrix A satisfies A ij *A jk =A ik Therefore, the criteria layer can be used to determine whether matrix A satisfies the conditions for a consistent matrix. This allows for verification of whether the criteria layer determines whether matrix A contains logical errors.

[0051] Typically, when performing consistency checks on the criterion-level judgment matrix A, it is first necessary to calculate the consistency index CI. The formula for calculating the consistency index CI is as follows:

[0052]

[0053] Where n represents the number of indicators, λ max Let λ be the largest eigenvalue in the criterion-layer judgment matrix A, and let λ be the largest eigenvalue in the matrix A. max The calculation formula is as follows:

[0054]

[0055] The eigenvalues ​​of each indicator in the criterion-level judgment matrix A are shown in Table 3.

[0056]

[0057] Therefore, as shown in Table 3, the largest eigenvalue λ of the criterion-level judgment matrix A is... max It is 4.00.

[0058] Then find the average random consistency index RI corresponding to the number of indicators. The values ​​of RI are shown in Table 4.

[0059] Table 4

[0060] n 1 2 3 4 5 6 7 8 9 10 11 RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.49 1.51

[0061] Finally, the consistency ratio CR is calculated using the consistency index CI and the average random consistency index RI. When CR < 0.1, the criterion-level judgment matrix is ​​considered consistent; otherwise, the values ​​of the criterion-level judgment matrix need to be readjusted. The formula for calculating the consistency ratio CR is as follows:

[0062]

[0063] Therefore, when the order of the criterion layer judgment matrix A is 4, RI = 0.90, and CR = 0.01204 is obtained by the above formula. At this time, CR is less than 0.1, and the eigenvector W can be used as the first weight vector for elimination, and the value of the criterion layer judgment matrix is ​​readjusted.

[0064] After readjusting the values ​​of the criterion-level judgment matrix and verifying its consistency, the geometric mean method can be used to determine the first weight of each indicator in the criterion-level judgment matrix, thus obtaining the first weight result.

[0065] First, calculate the nth root of the product of the elements in each row of the criterion-level judgment matrix A. The calculation formula is as follows:

[0066]

[0067] Among them, a ij Let be the element in the i-th row and j-th column of matrix A, and n represent the order of matrix A.

[0068] Next, the weights and eigenvectors of each index are calculated using the following formula: the nth root of each element in the i-th row divided by the sum of the nth roots of the products of the elements in each row of matrix A.

[0069]

[0070]

[0071] Finally, by normalizing the above results column by column, the first weight results of each indicator in the criterion layer judgment matrix can be obtained, as shown in Table 5.

[0072] Table 5

[0073]

[0074]

[0075] S1022, For each second file, establish a scheme-level judgment matrix and calculate the second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, and not pre-caching the second file.

[0076] Similarly, the calculation process in step S1022 above is similar, and the geometric mean method is used to determine the second weight of each scheme in the scheme layer judgment matrix to obtain the second weight result.

[0077] In addition, before calculating the second weight result of the scheme layer judgment matrix, it is necessary to perform a consistency check on the scheme layer judgment matrix. If the scheme layer judgment matrix passes the consistency check, the second weight result of the scheme layer judgment matrix is ​​calculated; if the scheme layer judgment matrix fails the consistency check, the value of the scheme layer judgment matrix is ​​readjusted.

[0078] S1023, Calculate the target score for each second document based on the first weight result and the second weight result.

[0079] Specifically, a weighted average can be used to calculate the target score for each second document.

[0080] S1024, sort the multiple second files according to the target score from largest to smallest, and take the first number of second files with the highest ranking as the first file, and pre-cache the first file to the target cache.

[0081] Typically, the target scores of the second files are sorted in ascending order. The higher the target score of the second file, the less necessary it is to cache. Therefore, the second files in the top 500 are used as the first files and are pre-cached in the target cache.

[0082] In order to enhance the availability of data system caching, the target cache in this application can be a distributed caching module based on ctg-cache.

[0083] Figure 3 A diagram illustrating an optional distributed caching module based on ctg-cache is shown. Figure 3 It is evident that the distributed caching module based on ctg-cache offers significant advantages in pre-caching small files compared to existing open-source caching components, including stronger scalability and higher stability. Therefore, in the event of various unexpected events, the application capabilities of the distributed caching module based on ctg-cache far exceed those of traditional caching products.

[0084] Step S104: When the target file is included in the target cache, the target file is sent back to the target device.

[0085] Step S106: If the target file is not included in the target cache, read the target file from the target data storage system, store the read target file in the target cache, and send the target file back to the target device.

[0086] According to an optional implementation of this application, in order to save storage resources in the data system, after storing the target file in the target cache, the first file with the lowest target score in the target cache can also be deleted from the target cache.

[0087] In this application, multiple second files in the target data storage system can be stored based on a hash structure. Compared with traditional data storage systems, the hash structure-based storage method is more conducive to flattened file management.

[0088] For example, Figure 4aThe diagram illustrates an alternative directory tree structure for a data system. In this system, to better manage file permissions, each directory is always stored in a nested hierarchy. This results in significant CPU consumption when querying a subdirectory. Especially for small files with lax permission control during temporary events, extremely high concurrency access will put immense pressure on the CPU, potentially causing system crashes.

[0089] To address the issues with the relevant data storage system, from a data structure perspective, the data system is a binary tree, and the structure for storing files is a hash table, thereby achieving flattened management of files. Figure 4b A schematic diagram of an optional hash storage structure is shown, which stores a large number of small files in a bucket. The hash value is calculated using UUID (Hash Function ForeSequence of Unique Ids) to find the hash pattern of the corresponding file, thereby supporting ultra-high concurrency access. At the same time, the storage method based on the hash structure has strong scalability.

[0090] In addition, in this application, during the process of writing files to disk, different levels of resource pools will be selected based on the target score of the files in order to save storage resources of the data system.

[0091] Figure 5 A schematic diagram of an optional storage resource pool is shown, in which the third file in multiple second files is stored in RAID1 (i.e., the first type of resource pool) of the target data storage system, and the fourth file in multiple second files is stored in RAID5 (i.e., the second type of resource pool) of the target data storage system.

[0092] Among them, the third file is a file whose target score in multiple second files is higher than the preset threshold, the fourth file is a file whose target score in multiple second files is not higher than the preset threshold, and the security level of the first type of resource pool is higher than the security level of the second type of resource pool.

[0093] It should be noted that the first quantity, preset threshold, etc., can be determined according to the specific needs of actual application. No restrictions are imposed here. The specific values ​​in the embodiments of this application are only used as examples for illustration.

[0094] In this embodiment, in response to an access request from the target device for accessing a target file, the target file is searched in the target cache. The target cache pre-caches a first number of first files, which are the files ranked highest in the target score among multiple second files in the target data storage system. The target score is a score determined based on the analytic hierarchy process (AHP) to reflect whether a second file should be pre-cached. If the target file is included in the target cache, it is sent back to the target device. If the target file is not included in the target cache, it is read from the target data storage system, stored in the target cache, and then sent back to the target device. The AHP-based target score ensures that the multiple second files in the target data storage system better meet business and performance requirements. This effectively improves the efficiency and stability of the target data storage system in reading files, thereby solving the technical problem of data storage systems easily experiencing lag and crashes under high-concurrency file access scenarios in related technologies.

[0095] Example 2

[0096] According to an embodiment of this application, a file reading device for implementing the file reading method in Embodiment 1 is also provided, such as... Figure 6 As shown, the file reading device includes at least a query module 61, a first feedback module 62, and a second feedback module 63, wherein:

[0097] The lookup module 61 is configured to, in response to an access request from the target device for accessing a target file, look up the target file from the target cache, wherein the target cache pre-caches a first number of first files, the first files being the files with the highest target scores among a plurality of second files in the target data storage system, and the target score being a score determined based on the analytic hierarchy process to reflect whether the second files are pre-cached.

[0098] According to an optional implementation of this application, before the lookup module 61 looks up the target file in the target cache, the target cache can also be determined in the following manner:

[0099] First, establish a criterion-level judgment matrix and calculate the first weight result of the criterion-level judgment matrix. The criterion-level judgment matrix shall include at least the following indicators: business importance of the file, user experience, recent access time and file size.

[0100] Optionally, before calculating the first weight result of the criterion layer judgment matrix, a consistency check can be performed on the criterion layer judgment matrix. If the criterion layer judgment matrix passes the consistency check, the first weight result of the criterion layer judgment matrix is ​​calculated; if the criterion layer judgment matrix fails the consistency check, the value of the criterion layer judgment matrix is ​​readjusted.

[0101] Specifically, the geometric mean method can be used to determine the first weight of each indicator in the criterion layer judgment matrix, thus obtaining the first weight result.

[0102] Next, for each second file, a scheme-level judgment matrix is ​​established, and the second weight result of the scheme-level judgment matrix is ​​calculated. The scheme-level judgment matrix includes the following schemes: pre-caching the second file, and not pre-caching the second file.

[0103] Optionally, before calculating the second weight result of the scheme-level judgment matrix, a consistency check can be performed on the scheme-level judgment matrix. If the scheme-level judgment matrix passes the consistency check, the second weight result of the scheme-level judgment matrix is ​​calculated; if the scheme-level judgment matrix fails the consistency check, the value of the scheme-level judgment matrix is ​​readjusted.

[0104] Specifically, the geometric mean method can be used to determine the second weight of each scheme in the scheme layer judgment matrix, thus obtaining the second weight result.

[0105] Then, the target score for each second file is calculated based on the first weighting result and the second weighting result.

[0106] Specifically, a weighted average can be used to calculate the target score for each second document.

[0107] Finally, the second files are sorted from largest to smallest based on the target score, and the first number of second files in the sorted list are used as the first file and pre-cached in the target cache.

[0108] The first feedback module 62 is used to directly feed back the target file to the target device when the target file is included in the target cache.

[0109] The second feedback module 63 is used to read the target file from the target data storage system when the target file is not included in the target cache, store the read target file in the target cache, and feed the target file back to the target device.

[0110] According to an optional implementation of this application, in order to save storage resources in the data system, after the second feedback module 63 stores the target file in the target cache, the first file with the lowest target score in the target cache can also be deleted from the target cache.

[0111] In this application, multiple second files in the target data storage system can be stored based on a hash structure. In order to solve the problems of related data storage systems, from the perspective of data structure, the data system is a binary tree, and the structure of storing files is a hash table, thereby realizing flat management of files.

[0112] In addition, in this application, during the process of writing files to disk, different levels of resource pools will be selected based on the target score of the files in order to save storage resources of the data system.

[0113] Specifically, the third file among the multiple second files is stored in the first type of resource pool in the target data storage system, and the fourth file among the multiple second files is stored in the second type of resource pool in the target data storage system. The third file is the file among the multiple second files whose target score is higher than a preset threshold, and the fourth file is the file among the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than that of the second type of resource pool.

[0114] It should be noted that each module in the file reading device in this embodiment corresponds one-to-one with each implementation step of the file reading method in Embodiment 1. Since Embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to Embodiment 1, and will not be elaborated further here.

[0115] Example 3

[0116] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored program, wherein the device where the non-volatile storage medium is located executes the file reading method in Embodiment 1 by running the program.

[0117] Specifically, the device containing the non-volatile storage medium executes the following steps by running this program: In response to an access request from the target device for accessing a target file, the target file is searched for in the target cache, wherein the target cache pre-caches a first number of first files, which are the files ranked highest in the target score among multiple second files in the target data storage system, and the target score is a score determined based on the analytic hierarchy process (AHP) to reflect whether to pre-cache the second files; if the target file is included in the target cache, the target file is sent back to the target device; if the target file is not included in the target cache, the target file is read from the target data storage system, stored in the target cache, and then sent back to the target device. This application solves the technical problem of data storage systems being prone to lag and crashes in high-concurrency file access scenarios in related technologies.

[0118] Optionally, before searching for the target file in the target cache, the method further includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: business importance of the file, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, or not pre-caching the second file; calculating a target score for each second file based on the first weight result and the second weight result; sorting multiple second files from largest to smallest based on the target score, and taking the first number of second files at the top of the sort as the first file, and pre-caching the first file to the target cache.

[0119] Optionally, before calculating the first weight result of the criterion-level judgment matrix, the method further includes: performing a consistency check on the criterion-level judgment matrix; calculating the first weight result of the criterion-level judgment matrix when the criterion-level judgment matrix passes the consistency check; and readjusting the values ​​of the criterion-level judgment matrix when the criterion-level judgment matrix fails the consistency check. Before calculating the second weight result of the scheme-level judgment matrix, the method further includes: performing a consistency check on the scheme-level judgment matrix; calculating the second weight result of the scheme-level judgment matrix when the scheme-level judgment matrix passes the consistency check; and readjusting the values ​​of the scheme-level judgment matrix when the scheme-level judgment matrix fails the consistency check.

[0120] Optionally, the multiple second files in the target data storage system are stored based on a hash structure; the third file among the multiple second files is stored in the first type of resource pool in the target data storage system, and the fourth file among the multiple second files is stored in the second type of resource pool in the target data storage system. The third file is a file among the multiple second files whose target score is higher than a preset threshold, and the fourth file is a file among the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than that of the second type of resource pool.

[0121] Optionally, the target cache is a distributed caching module based on ctg-cache.

[0122] Example 4

[0123] According to an embodiment of this application, a processor is also provided for running a program, wherein the program executes the file reading method in embodiment 1 during runtime.

[0124] Specifically, the program executes the following steps during runtime: In response to an access request from the target device for accessing a target file, it searches for the target file in the target cache, wherein the target cache pre-caches a first number of first files, which are the files ranked highest in the target score among multiple second files in the target data storage system, and the target score is a score determined based on the analytic hierarchy process (AHP) to reflect whether to pre-cache the second files; if the target file is included in the target cache, it is sent back to the target device; if the target file is not included in the target cache, it is read from the target data storage system, stored in the target cache, and sent back to the target device. This application solves the technical problem of data storage systems being prone to lag and crashes in high-concurrency file access scenarios in related technologies.

[0125] Optionally, before searching for the target file in the target cache, the method further includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: business importance of the file, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, or not pre-caching the second file; calculating a target score for each second file based on the first weight result and the second weight result; sorting multiple second files from largest to smallest based on the target score, and taking the first number of second files at the top of the sort as the first file, and pre-caching the first file to the target cache.

[0126] Optionally, before calculating the first weight result of the criterion-level judgment matrix, the method further includes: performing a consistency check on the criterion-level judgment matrix; calculating the first weight result of the criterion-level judgment matrix when the criterion-level judgment matrix passes the consistency check; and readjusting the values ​​of the criterion-level judgment matrix when the criterion-level judgment matrix fails the consistency check. Before calculating the second weight result of the scheme-level judgment matrix, the method further includes: performing a consistency check on the scheme-level judgment matrix; calculating the second weight result of the scheme-level judgment matrix when the scheme-level judgment matrix passes the consistency check; and readjusting the values ​​of the scheme-level judgment matrix when the scheme-level judgment matrix fails the consistency check.

[0127] Optionally, the multiple second files in the target data storage system are stored based on a hash structure; the third file among the multiple second files is stored in the first type of resource pool in the target data storage system, and the fourth file among the multiple second files is stored in the second type of resource pool in the target data storage system. The third file is a file among the multiple second files whose target score is higher than a preset threshold, and the fourth file is a file among the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than that of the second type of resource pool.

[0128] Optionally, the target cache is a distributed caching module based on ctg-cache.

[0129] Example 5

[0130] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the file reading method of embodiment 1 through the computer program.

[0131] Specifically, the processor is configured to execute the following steps via a computer program: in response to an access request from a target device for accessing a target file, searching for the target file in a target cache, wherein the target cache pre-caches a first number of first files, the first files being the files ranked highest in a target score among multiple second files in the target data storage system, and the target score being a score determined based on the analytic hierarchy process (AHP) to reflect whether to pre-cache the second files; if the target file is included in the target cache, sending the target file back to the target device; if the target file is not included in the target cache, reading the target file from the target data storage system, storing the read target file in the target cache, and sending the target file back to the target device. This application solves the technical problem of data storage systems being prone to lag and crashes in high-concurrency file access scenarios in related technologies.

[0132] Optionally, before searching for the target file in the target cache, the method further includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: business importance of the file, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, or not pre-caching the second file; calculating a target score for each second file based on the first weight result and the second weight result; sorting multiple second files from largest to smallest based on the target score, and taking the first number of second files at the top of the sort as the first file, and pre-caching the first file to the target cache.

[0133] Optionally, before calculating the first weight result of the criterion-level judgment matrix, the method further includes: performing a consistency check on the criterion-level judgment matrix; calculating the first weight result of the criterion-level judgment matrix when the criterion-level judgment matrix passes the consistency check; and readjusting the values ​​of the criterion-level judgment matrix when the criterion-level judgment matrix fails the consistency check. Before calculating the second weight result of the scheme-level judgment matrix, the method further includes: performing a consistency check on the scheme-level judgment matrix; calculating the second weight result of the scheme-level judgment matrix when the scheme-level judgment matrix passes the consistency check; and readjusting the values ​​of the scheme-level judgment matrix when the scheme-level judgment matrix fails the consistency check.

[0134] Optionally, the multiple second files in the target data storage system are stored based on a hash structure; the third file among the multiple second files is stored in the first type of resource pool in the target data storage system, and the fourth file among the multiple second files is stored in the second type of resource pool in the target data storage system. The third file is a file among the multiple second files whose target score is higher than a preset threshold, and the fourth file is a file among the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than that of the second type of resource pool.

[0135] Optionally, the target cache is a distributed caching module based on ctg-cache.

[0136] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0142] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A file reading method, characterized in that, include: In response to an access request from a target device for accessing a target file, the target file is retrieved from a target cache, wherein a first number of first files are pre-cached in the target cache, the first files being the files with the highest target scores among a plurality of second files in the target data storage system, and the target score being a score determined based on the analytic hierarchy process to reflect whether the second files are pre-cached; When the target file is included in the target cache, the target file is sent back to the target device; If the target file is not included in the target cache, the target file is read from the target data storage system, stored in the target cache, and then sent back to the target device. The method for determining the first file in the target cache includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: business importance of the file, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, or not pre-caching the second file; calculating the target score for each second file based on the first weight result and the second weight result; sorting multiple second files from largest to smallest based on the target score, and selecting the first number of second files at the top of the sort as the first file, and pre-caching the first file in the target cache.

2. The method according to claim 1, characterized in that, Before calculating the first weight result of the criterion layer judgment matrix, the method further includes: performing a consistency check on the criterion layer judgment matrix; when the criterion layer judgment matrix passes the consistency check, calculating the first weight result of the criterion layer judgment matrix; when the criterion layer judgment matrix fails the consistency check, readjusting the values ​​of the criterion layer judgment matrix. Before calculating the second weight result of the scheme layer judgment matrix, the method further includes: performing a consistency check on the scheme layer judgment matrix; when the scheme layer judgment matrix passes the consistency check, calculating the second weight result of the scheme layer judgment matrix; when the scheme layer judgment matrix fails the consistency check, readjusting the value of the scheme layer judgment matrix.

3. The method according to claim 1, characterized in that, Calculating the first weight result of the criterion layer judgment matrix includes: determining the first weight of each indicator in the criterion layer judgment matrix using the geometric mean method, and obtaining the first weight result; Calculating the second weight result of the scheme layer judgment matrix includes: determining the second weight of each scheme in the scheme layer judgment matrix using the geometric mean method, and obtaining the second weight result.

4. The method according to claim 1, characterized in that, After storing the target file in the target cache, the method further includes: Remove the first file with the lowest target score from the target cache.

5. The method according to any one of claims 1 to 4, characterized in that, The second files in the target data storage system are stored based on a hash structure; The third files in the multiple second files are stored in a first type of resource pool in the target data storage system, and the fourth files in the multiple second files are stored in a second type of resource pool in the target data storage system. The third files are files in the multiple second files whose target score is higher than a preset threshold, and the fourth files are files in the multiple second files whose target score is not higher than the preset threshold. The security level of the first type of resource pool is higher than the security level of the second type of resource pool.

6. The method according to any one of claims 1 to 4, characterized in that, The target cache is a distributed cache module based on ctg-cache.

7. A file reading device, characterized in that, include: A lookup module is configured to, in response to an access request from a target device for accessing a target file, search for the target file in a target cache. The target cache contains a first number of first files, which are files ranked highest in a target score among multiple second files in a target data storage system. The target score is a score determined based on the Analytic Hierarchy Process (AHP) to reflect whether to pre-cache the second files. The method for determining the first files in the target cache includes: establishing a criterion-level judgment matrix and calculating a first weight result of the criterion-level judgment matrix, wherein the criterion-level judgment matrix includes at least the following indicators: file business importance, user experience, recent access time, and file size; for each second file, establishing a scheme-level judgment matrix and calculating a second weight result of the scheme-level judgment matrix, wherein the scheme-level judgment matrix includes the following schemes: pre-caching the second file, not pre-caching the second file; calculating the target score for each second file based on the first weight result and the second weight result; sorting the multiple second files from largest to smallest based on the target score, and pre-caching the first number of second files ranked highest as the first files in the target cache. The first feedback module is used to directly feed back the target file to the target device when the target cache includes the target file; The second feedback module is used to read the target file from the target data storage system when the target file is not included in the target cache, store the read target file in the target cache, and feed the target file back to the target device.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the device containing the non-volatile storage medium executes the file reading method according to any one of claims 1 to 6 by running the program.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the file reading method of any one of claims 1 to 6 through the computer program.

Images