File processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a file processing method and device, equipment, a storage medium and a program product, and the method specifically comprises the steps that a downloading request of a to-be-downloaded file is received, and the downloading request comprises file elements of the to-be-downloaded file; in response to the downloading request, querying file elements of a to-be-downloaded file by using a Bloom filter to obtain a first query result; wherein the configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion state value of the Bloom filter, and the system fusion state value represents the health state of the Bloom filter; when the first query result represents that the file state of the to-be-downloaded file is a downloaded state, querying file elements of the to-be-downloaded file in a database to obtain a second query result; and based on the second query result, executing a processing operation on the to-be-downloaded file. According to the embodiment of the invention, the misjudgment rate of the Bloom filter can be effectively reduced, so that repeated downloading of the file can be effectively avoided.

Description

Technical Field

[0001] This application belongs to the field of transmission and transmission technology, and particularly relates to a method, apparatus, device, computer-readable storage medium, and computer program product for document processing. Background Technology

[0002] In real-world scenarios, efficient file processing is crucial for ensuring business continuity and system performance. Among these, avoiding redundant file processing (such as repeated downloads and storage) is intuitively important.

[0003] Taking file download as an example, existing technologies often use Bloom filters as a pre-deduplication mechanism to quickly determine whether a file already exists, thus avoiding duplicate downloads. However, as file downloads continue, the number of elements stored in the Bloom filter accumulates, and the false positive rate of the Bloom filter is directly proportional to the number of elements it stores. Therefore, as files are continuously downloaded, the false positive rate of the Bloom filter also increases, leading to a serious problem: when a file that has already been downloaded is mistakenly identified as "non-existent" by the Bloom filter, the system will trigger a duplicate download of that file. Summary of the Invention

[0004] This application provides a method, apparatus, device, computer-readable storage medium, and computer program product for file processing, which can effectively reduce the false positive rate of Bloom filters and thus effectively avoid duplicate file restrictions.

[0005] In a first aspect, embodiments of this application provide a document processing method, the method comprising: Receive a download request for a file to be downloaded, the download request including the file elements of the file to be downloaded; In response to a download request, a Bloom filter is used to query the file elements of the file to be downloaded to obtain the first query result. The configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion status value of the Bloom filter, which represents the health status of the Bloom filter. If the first query result indicates that the file status of the file to be downloaded is "downloaded", then the file elements of the file to be downloaded are queried in the database to obtain the second query result; Based on the second query result, perform the processing operation on the file to be downloaded.

[0006] Secondly, embodiments of this application provide a document processing apparatus, the apparatus comprising: The receiving module is used to receive download requests, which include file elements of the file to be downloaded. The query module is used to respond to download requests by using a Bloom filter to query the file elements of the file to be downloaded and obtain the first query result. The configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion status value of the Bloom filter, which represents the health status of the Bloom filter. The query module is also used to query the file elements of the file to be downloaded in the database to obtain a second query result when the first query result indicates that the file status of the file to be downloaded is already downloaded. The execution module is used to perform processing operations on the files to be downloaded based on the second query result.

[0007] Thirdly, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for reading and executing the computer program instructions stored in the memory to perform the file processing method provided in the first aspect.

[0008] Fourthly, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the file processing method provided in the first aspect is implemented.

[0009] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the file processing method provided in the first aspect.

[0010] In this embodiment, after receiving a download request for a file to be downloaded, a Bloom filter is first used to query the file elements of the file to be downloaded in the download request to determine whether the file has already been downloaded. Even if the first query result output by the Bloom filter indicates that the file has been downloaded, considering the potential for false positives in Bloom filters, the file elements of the file to be downloaded can be further queried in the database to obtain a more accurate second query result. Furthermore, the configuration parameters of the Bloom filter can be dynamically adjusted based on system fusion status values ​​that characterize the health of the Bloom filter. This effectively reduces the false positive rate of the Bloom filter and allows for verification in the database, thereby effectively preventing duplicate downloads of files. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the framework of a document processing method provided in an embodiment of this application; Figure 2 This is one of the flowcharts illustrating a document processing method provided in an embodiment of this application; Figure 3 This is a second schematic flowchart of a document processing method provided in an embodiment of this application; Figure 4 This is a third flowchart illustrating a document processing method provided in an embodiment of this application; Figure 5 This is a fourth flowchart illustrating a document processing method provided in an embodiment of this application; Figure 6 This is the fifth flowchart illustrating a document processing method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a document processing apparatus provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0015] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0016] To address the technical problems existing in the prior art, embodiments of this application provide a method, apparatus, device, computer-readable storage medium, and computer program product for file processing. Upon receiving a download request for a file to be downloaded, a Bloom filter is first used to query the file elements of the file to be downloaded in the download request to determine whether the file has already been downloaded. Even if the first query result output by the Bloom filter indicates that the file has been downloaded, considering the potential for false positives in Bloom filters, the file elements of the file to be downloaded can be further queried in a database to obtain a more accurate second query result. Furthermore, the configuration parameters of the Bloom filter can be dynamically adjusted based on system fusion status values ​​that characterize the health of the Bloom filter. This effectively reduces the false positive rate of the Bloom filter, and the database can be used for further verification, thereby effectively preventing duplicate downloads of files.

[0017] It should be noted that the file processing method provided in this application embodiment can be executed by a file processing device or a control module in the file processing device for executing the file processing method. In this application embodiment, the file processing method provided in this application embodiment is described in detail using the file processing device executing the file processing method as an example.

[0018] First, to facilitate understanding of the document processing method provided in the embodiments of this application, before introducing the document processing method provided in the embodiments of this application, let's first combine it with... Figure 1 The framework involved in the document processing method provided in the embodiments of this application will be described in detail.

[0019] Figure 1 This is a schematic diagram of the framework of a file processing method provided in an embodiment of this application.

[0020] like Figure 1 As shown in the embodiments of this application, a file processing method mainly includes the following parts: Adjusting the parameters of the Bloom filter is crucial because as the number of downloaded files increases, the number of file elements in the Bloom filter also increases, which can lead to a higher false positive rate. Therefore, it is necessary to monitor and statistically analyze the data to dynamically adjust the parameters of the Bloom filter in real time, thereby reducing the false positive rate to an expected level.

[0021] The file processing method provided in this application can dynamically adjust the element threshold of the Bloom filter. While ensuring a controllable false positive rate, it minimizes system resource consumption, thereby adapting to different application scenarios. Under high load, high frequency access, or high false positive rate conditions, the element threshold is increased to improve performance; under low load, low frequency access, or low false positive rate conditions, the element threshold is decreased to save resources.

[0022] For file deduplication, upon receiving a file download request, file elements are extracted, and a Bloom filter is queried to confirm whether the file has already been downloaded. If the filter returns that the file element does not exist, the file has not been downloaded, and downloading can proceed. If the Bloom filter returns that the file element exists, due to the potential for false positives, the cache and database must be consulted to ultimately confirm whether the file has been downloaded. If it has not been downloaded, downloading is still performed.

[0023] Cache management is necessary because Bloom filters can make false judgments. If the Bloom filter returns that the file exists, but the database still needs to be queried for final confirmation, a cache is introduced to reduce the frequency of database queries. By monitoring and statistically analyzing all falsely judged files, file elements of hot files that meet certain conditions are cached, thereby improving the performance of deduplication verification.

[0024] Based on the above, this application provides a file processing method. The file processing method provided by this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Figure 2 This is one of the flowcharts illustrating a file processing method provided in this application embodiment.

[0026] like Figure 2 As shown, the execution subject of the file processing method can be a file processing device. Based on this, the file processing method can specifically include the following steps: S210 receives a download request for the file to be downloaded.

[0027] The file processing apparatus is capable of receiving a download request for a file to be downloaded. This download request can be used to trigger the download process for the file. In some embodiments, the download request may include file elements of the file to be downloaded, wherein the file elements may be relevant information used to uniquely identify the file to be downloaded, such as a file identifier, file content keywords, a file hash value, and file metadata, etc., without specific limitations here.

[0028] S220, in response to the download request, uses a Bloom filter to query the file elements of the file to be downloaded and obtains the first query result.

[0029] Specifically, after receiving a download request for a file to be downloaded, the file processing device can use a Bloom filter to query the file elements of the file to be downloaded in order to avoid the file being downloaded repeatedly, and obtain a first query result.

[0030] The first query result mentioned above can be used to characterize the file status of the file to be downloaded. The file status of the file to be downloaded includes a downloaded status or a not downloaded status, which is not specifically limited here.

[0031] More specifically, a Bloom filter can store multiple first file elements. Thus, when querying the file elements of the file to be downloaded using the Bloom filter, if the multiple first file elements stored in the Bloom filter include the file elements of the file to be downloaded, the first query result can indicate that the file status of the file to be downloaded is downloaded. If the multiple first file elements stored in the Bloom filter do not include the file elements of the file to be downloaded, the first query result can indicate that the file status of the file to be downloaded is not downloaded.

[0032] In this embodiment, the system fusion state value of the Bloom filter can characterize the health status of the Bloom filter. Specifically, the higher the system fusion state value, the worse the health status of the Bloom filter and the greater the risk of misjudgment, and vice versa. Therefore, the configuration parameters of the Bloom filter can be dynamically adjusted based on the system fusion state value of the Bloom filter, thereby effectively reducing the risk of misjudgment of the Bloom filter.

[0033] S230, if the first query result indicates that the file status of the file to be downloaded is already downloaded, the database is queried for the file elements of the file to be downloaded to obtain the second query result.

[0034] Considering the inherent risk of false positives with Bloom filters, the file processing device, after obtaining the first query result (which indicates the file status of the file to be downloaded is already downloaded), can continue querying the file elements of the file to be downloaded in the database to obtain the corresponding second query result. Furthermore, the aforementioned database can be a MySQL database.

[0035] The second query result can also be used to characterize the file status of the file to be downloaded. More specifically, the database can store multiple second file elements, all of which are file elements of downloaded files. These multiple second file elements can include the aforementioned multiple first file elements. Thus, if the file element of the file to be downloaded can be found among the multiple second file elements stored in the database, it indicates that the file to be downloaded has been downloaded, and the obtained second query result can characterize the file status of the file to be downloaded as downloaded. Conversely, if the file element of the file to be downloaded cannot be found among the multiple second file elements stored in the database, it indicates that the file to be downloaded has not been downloaded, and the obtained second query result can characterize the file status of the file to be downloaded as not downloaded.

[0036] It should also be noted that if the first query result indicates that the file to be downloaded is in an undownloaded state, the file can be downloaded directly.

[0037] S240, based on the second query result, perform the processing operation on the file to be downloaded.

[0038] Specifically, if the second query result indicates that the file status of the file to be downloaded is "downloaded", then the download request for the file to be downloaded is ignored; otherwise, if the second query result indicates that the file status of the file to be downloaded is "not downloaded", then the file to be downloaded is downloaded.

[0039] It should be noted that if a file to be downloaded needs to be downloaded, after the file to be downloaded is downloaded, the relevant information of the file to be downloaded (filename, creation time, update time, MD5) will be stored in the database. By listening to the binlog log of the database, the file elements of the file to be downloaded will be automatically updated in the Bloom filter using the Redis Boolmfilter plugin.

[0040] In this embodiment, after receiving a download request for a file to be downloaded, a Bloom filter is first used to query the file elements of the file to be downloaded in the download request to determine whether the file has already been downloaded. Even if the first query result output by the Bloom filter indicates that the file has been downloaded, considering the potential for false positives in Bloom filters, the file elements of the file to be downloaded can be further queried in the database to obtain a more accurate second query result. Furthermore, the configuration parameters of the Bloom filter can be dynamically adjusted based on system fusion status values ​​that characterize the health of the Bloom filter. This effectively reduces the false positive rate of the Bloom filter and allows for verification in the database, thereby effectively preventing duplicate downloads of files.

[0041] Since the configuration parameters of the Bloom filter can be dynamically adjusted based on the system fusion state value of the Bloom filter in the aforementioned embodiments, in order to accurately adjust the configuration parameters of the Bloom filter and thus effectively reduce the false positive rate of the Bloom filter, in one embodiment, such as Figure 3 As shown, prior to S220, the document processing method provided in this application embodiment may specifically include the following steps: S310, determine the system fusion state value of the Bloom filter.

[0042] S320 dynamically adjusts the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter.

[0043] It should be noted that different system fusion state values ​​correspond to different parameter adjustment strategies, which are not specifically limited here.

[0044] Specifically, the system fusion state value of the Bloom filter can be determined. Since the system fusion state value can characterize the health status of the Bloom filter, the configuration parameters of the Bloom filter can be dynamically adjusted according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter based on the different health statuses represented by different system fusion state values. This can reduce the false judgment rate of the Bloom filter and improve its health status.

[0045] In this embodiment, by determining the system fusion state value used to characterize the health status of the Bloom filter, and by dynamically adjusting the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter, the false judgment rate of the Bloom filter can be accurately and effectively reduced, thereby effectively avoiding duplicate downloads of files.

[0046] In order to accurately obtain the system fusion state value of the Bloom filter, in one embodiment, the above-mentioned S310 may specifically include the following steps: Monitor the first metric corresponding to the Bloom filter, which includes the number of file elements stored in the Bloom filter and the false positive rate of file elements; The system fusion state value of the Bloom filter is calculated based on the number of file elements and the false positive rate of file elements.

[0047] As the number of first file elements stored in the Bloom filter gradually increases, the false positive rate of the Bloom filter also increases. Therefore, the number of first file elements stored in the Bloom filter can, to some extent, reflect the false positive rate of the Bloom filter. The file element false positive rate of the Bloom filter can intuitively reflect its false positive rate. Therefore, in some embodiments, the first indicator may include the number of file elements stored in the Bloom filter and the file element false positive rate. The file element false positive rate may include the actual false positive rate, the random false positive rate, or even a combined false positive rate based on both the actual and random false positive rates. No specific limitation is made here.

[0048] Specifically, it is possible to monitor the first indicator corresponding to the Bloom filter. Since the first indicator can include relevant information that reflects the misjudgment situation of the Bloom filter, such as the number of file elements stored in the Bloom filter and the file element misjudgment rate, it is possible to calculate the system fusion state value of the Bloom filter based on the number of file elements and the file element misjudgment rate.

[0049] In one example, if the file element misjudgment rate includes the file element random misjudgment rate, then the file element random misjudgment rate can be expressed by the following formula (1). This formula reflects the probability that a certain file element is incorrectly judged to exist under random conditions.

[0050] (1) in, The number of file elements stored in the Bloom filter for the multiple first file elements. The size of the bit array. The number of hash functions used is e, which is a natural constant, approximately equal to 2.71828.

[0051] If the file false positive rate includes the actual false positive rate of file elements, the formula for calculating the actual false positive rate of file elements can be as follows: Formula (2) (2) in, This represents the number of file elements falsely identified by the Bloom filter. This represents the number of first file elements stored in the Bloom filter, i.e., the total number.

[0052] If the file element false positive rate includes the overall false positive rate, then the overall false positive rate (Average False Positive Rate, AFPR) can be expressed as shown in formula (3): (3) Wherein, AFPR represents the overall false positive rate. Weights for identifying the random misclassification rate of file elements. This represents the actual false positive rate for file elements. The weight representing the actual misclassification rate of file elements.

[0053] Additionally, it should be noted that the configuration items in the Bloom filter's parameter configuration file can include file element threshold, file element random false positive rate threshold, file element actual false positive rate threshold, random false positive rate weight, actual false positive rate weight, element weight, overall false positive rate threshold, expected false positive rate, false positive monitoring time, system fusion status value, system fusion status threshold, dynamic adjustment strategy, and initial fusion status threshold. Among these, the file element threshold defines the maximum value of an element in the Bloom filter; the file element random false positive rate threshold defines the maximum random false positive rate of the Bloom filter; the file element actual false positive rate threshold defines the maximum allowed actual false positive rate of the Bloom filter within a certain time range; the random false positive rate weight indicates the proportion of the random false positive rate in calculating the overall false positive rate; the actual false positive rate weight represents the proportion of the actual false positive rate in calculating the overall false positive rate; the overall false positive rate threshold indicates the maximum allowed overall false positive rate of the Bloom filter; the expected false positive rate indicates the expected false positive rate of the Bloom filter; and the false positive monitoring time defines the calculation standard for the actual false positive rate from a time perspective.

[0054] In this embodiment, the false positive risk of the Bloom filter is comprehensively assessed from different dimensions by monitoring the number of document elements and the false positive rate in the Bloom filter. Based on these two indicators, a system fusion state value that accurately characterizes the health status or credibility of the Bloom filter can be calculated. Then, the parameters can be dynamically adjusted through this system fusion state value to effectively reduce the false positive rate of the Bloom filter.

[0055] In order to accurately and comprehensively describe the file processing method provided in the embodiments of this application, in one embodiment, the step of calculating the system fusion state value of the Bloom filter based on the number of file elements and the file element false positive rate may specifically include the following steps: Obtain the first weight information; Based on the first and second weights, the number of file elements and the file element misclassification rate are weighted and summed to obtain the system fusion state value of the Bloom filter.

[0056] In some embodiments, the first weight information includes a first weight for measuring the importance of the number of file elements and a second weight for measuring the importance of the false positive rate of file elements. Specifically, a larger first weight indicates that the number of file elements is more important in calculating the system fusion state value of the Bloom filter, and vice versa. A smaller first weight indicates that the number of file elements is less important in calculating the system fusion state value of the Bloom filter. The second weight is analogous to the first weight and will not be elaborated further here.

[0057] Specifically, it is possible to obtain first weight information, which may include a first weight for measuring the importance of the number of file elements and a second weight for measuring the importance of the file element misclassification rate. Based on the first weight and the second weight of the first weight information, the number of file elements and the file element misclassification rate can be weighted and summed to obtain the system fusion state value of the Bloom filter.

[0058] In this embodiment, by obtaining first weight information and performing weighted summation on the number of file elements and the false positive rate of file elements based on the obtained first weight information, the system fusion state value of the Bloom filter can be accurately obtained. Then, the configuration parameters of the Bloom filter can be accurately adjusted based on the system fusion state value to effectively reduce the false positive rate of the Bloom filter.

[0059] Based on this, in order to describe in detail the file processing method provided in the embodiments of this application, in one embodiment, before the step of performing weighted summation processing on the number of file elements and the file element misjudgment rate based on the first weight and the second weight to obtain the system fusion state value of the Bloom filter, the file processing method provided in the embodiments of this application may further include the following steps: Obtain the first mapping relationship between the number of elements and the score, and the second mapping relationship between the false judgment rate and the score; Based on the first mapping relationship, determine the first score corresponding to the number of file elements; Based on the second mapping relationship, determine the second score corresponding to the misjudgment rate of file elements; Based on this, the steps described above, which involve weighted summation of the number of file elements and the false positive rate of file elements based on the first and second weights to obtain the system fusion state value of the Bloom filter, may specifically include: Based on the first weight and the second weight, the first score and the second score are weighted and summed to obtain the system fusion state value of the Bloom filter.

[0060] Specifically, after obtaining the first weight information, it is possible to obtain a first mapping relationship between the number of elements and the score, and a second mapping relationship between the misclassification rate and the score. Based on the first mapping relationship, it is possible to determine the first score corresponding to the number of file elements, and based on the second mapping relationship, it is possible to determine the second score corresponding to the misclassification rate of file elements. Thus, when calculating the system fusion state value, the first score and the second score are weighted and summed based on the first and second weights to obtain the system fusion state value of the Bloom filter.

[0061] In one example, if the above file element misjudgment rate includes the random misjudgment rate of file elements and the actual misjudgment rate of file elements, then the above second mapping relationship can include the first sub-mapping relationship between the random misjudgment rate and the score and the second sub-mapping relationship between the actual misjudgment rate and the score. The second score corresponding to the file element misjudgment rate can include the first sub-score of the random misjudgment rate of file elements and the second sub-score corresponding to the actual misjudgment rate of file elements.

[0062] Thus, based on the first weight and the second weight, the first score and the second score are weighted and summed to obtain the system fusion state value of the Bloom filter, which can satisfy the following formula (4): (4) in, The first score corresponds to the number of file elements in the Bloom filter. The first sub-score representing the random misclassification rate of file elements. The second sub-score represents the actual misclassification rate of file elements. These represent the weights of the corresponding indicators.

[0063] Based on this, the system fusion state value of the Bloom filter can be normalized to obtain the normalized system fusion state value. Specifically, it can be shown in the following formula (5): (5) It should be noted that the scoring criteria for the above indicators can be specified based on the corresponding thresholds in the configuration file. For example, the random false positive rate of file elements: if the threshold for the random false positive rate of configuration file elements in the configuration file is 5%, then the following scoring criteria can be specified:

[0064] Other indicators are similar and will not be elaborated upon here.

[0065] In this embodiment, by obtaining the mapping relationship between each monitoring indicator and the score, and by determining the score corresponding to each monitoring indicator through the mapping relationship, the normalization processing of the multiple monitoring indicators is realized, and the scores of each monitoring indicator are weighted and summed to accurately obtain the system fusion state of the Bloom filter.

[0066] To describe in detail the file processing method provided in the embodiments of this application, in one embodiment, before dynamically adjusting the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter, the file processing method provided in the embodiments of this application further includes: If the system fusion status value is greater than the initialization threshold, determine the initialization parameters used to adjust the configuration parameters of the Bloom filter; The configuration parameters of the Bloom filter are dynamically adjusted to the initialization parameters.

[0067] The initialization threshold can be a threshold used to trigger the initialization of the Bloom filter; however, the specific value of this initialization threshold is not specifically limited in this embodiment. Furthermore, the aforementioned initialization parameters can be relevant parameters used to initialize the Bloom filter, and are not specifically limited here.

[0068] Specifically, when the calculated system fusion state value is greater than the initialization threshold, the initialization parameters for adjusting the configuration parameters of the Bloom filter can be determined. Then, according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter, the configuration parameters of the Bloom filter can be dynamically adjusted to the initialization parameters, thereby realizing the initialization of the Bloom filter.

[0069] In this embodiment, when the system fusion status value is greater than the initialization threshold, indicating that the health status of the Bloom filter is poor, the initialization parameters can be determined, and the configuration parameters of the Bloom filter can be dynamically adjusted to the initialization parameters to achieve the initialization of the Bloom filter, thereby effectively reducing the false judgment rate of the Bloom filter.

[0070] In another embodiment, before dynamically adjusting the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter, the document processing method provided in this application embodiment further includes: If the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, determine the parameter adjustment strategy corresponding to the system fusion status value. When the parameter adjustment strategy corresponding to the system fusion state value is the optimization adjustment strategy, determine the target configuration parameters for adjusting the configuration parameters of the Bloom filter; Dynamically adjust the configuration parameters of the Bloom filter to the target configuration parameters.

[0071] The aforementioned parameter configuration threshold can be a relevant threshold used to trigger the adjustment of the Bloom filter configuration parameters. This parameter configuration threshold is less than the initialization threshold. In this embodiment of the application, the specific value of the parameter configuration threshold is not specifically limited.

[0072] In some embodiments, the target configuration parameters described above include at least one of a bit array scaling factor and a function number scaling factor.

[0073] Specifically, when the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, indicating that the Bloom filter is in poor health, the system can determine the parameter adjustment strategy corresponding to the system fusion status value. Furthermore, when the parameter adjustment strategy corresponding to the system fusion status value is an optimization adjustment strategy, the system can determine the target configuration parameters for adjusting the configuration parameters of the Bloom filter. The system can also dynamically adjust the configuration parameters of the Bloom filter to the target configuration parameters according to the parameter adjustment strategy corresponding to the system fusion status value of the Bloom filter, thereby achieving the optimization of the Bloom filter.

[0074] In this embodiment, when the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, indicating that the health status of the Bloom filter is poor, the system can determine the parameter adjustment strategy corresponding to the system fusion status value. If the parameter adjustment strategy is an optimization adjustment strategy, the target configuration parameter can be obtained, and then the configuration parameter of the Bloom filter can be dynamically adjusted to the target configuration parameter. This achieves the optimization of the Bloom filter and effectively reduces the false positive rate of the Bloom filter.

[0075] Therefore, in order to effectively optimize the Bloom filter, in one embodiment, the step of dynamically adjusting the configuration parameters of the Bloom filter to the target configuration parameters may specifically include: A target Bloom filter is constructed based on the target configuration parameters; Update the file elements stored in the database to the target Bloom filter; Update the target Bloom filter to a Bloom filter.

[0076] Specifically, after obtaining the target configuration parameters, a target Bloom filter can be constructed based on the target configuration parameters, and the file elements stored in the database can be updated to the target Bloom filter, thereby updating the target Bloom filter to a Bloom filter.

[0077] In one example, if the target configuration parameter mentioned above can include a bit array scaling factor, that is, increasing the bit array length based on the target configuration parameter, for example, if the configured bit array scaling factor is 2.0, then a new Bloom filter is constructed, and the bit array length changes from m to 2m. Next, file elements stored in the database, i.e., downloaded file records, can be read and synchronized to the new Bloom filter, i.e., the target Bloom filter. After the file elements stored in the database are synchronized, the system switches to the target Bloom filter, and the Bloom filter is deactivated. Through updating the Bloom filter, the bit array length of the Bloom filter is dynamically adjusted.

[0078] In another example, a new Bloom filter is constructed based on the function number scaling factor in the configuration file. Then, file elements stored in the database, i.e., downloaded file records, can be read and synchronized to the new Bloom filter, i.e., the target Bloom filter. After the file elements stored in the database are synchronized, the target Bloom filter is switched to and the Bloom filter is deactivated. By updating the Bloom filter, the number of functions can be dynamically adjusted.

[0079] In this embodiment, a target Bloom filter can be constructed based on the acquired target configuration parameters, and the file elements stored in the Bloom filter can be updated to the target Bloom filter, and the target Bloom filter can be updated to a Bloom filter. In this way, the optimization of the Bloom filter can be achieved accurately and effectively.

[0080] In another embodiment, the document processing method provided in this application further includes: If the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, determine the parameter adjustment strategy corresponding to the system fusion status value. When the parameter adjustment strategy corresponding to the system fusion status value is the initialization adjustment strategy, determine the initialization configuration parameters used to adjust the configuration parameters of the Bloom filter; The configuration parameters of the Bloom filter are dynamically adjusted to the initialization parameters.

[0081] Specifically, when the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, the parameter adjustment strategy corresponding to the system fusion status value is determined. If the parameter adjustment strategy is the initialization adjustment strategy, the initialization configuration parameter used to adjust the configuration parameter of the Bloom filter is determined. Then, the configuration parameter of the Bloom filter can be dynamically adjusted to the initialization parameter to realize the initialization of the Bloom filter.

[0082] In this embodiment, when the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, indicating that the health status of the Bloom filter is poor, the system can determine the parameter adjustment strategy corresponding to the system fusion status value. If the parameter adjustment strategy is the initialization adjustment strategy, the target configuration parameter can be obtained, and then the configuration parameter of the Bloom filter can be dynamically adjusted to the target configuration parameter. This achieves the initialization of the Bloom filter and effectively reduces the false positive rate of the Bloom filter.

[0083] To provide a comprehensive and detailed description of the file processing method provided in the embodiments of this application, in one embodiment, the file processing method provided in the embodiments of this application further includes: Monitor the second metric corresponding to the Bloom filter. The second metric includes the file deduplication information of the Bloom filter and the system load information of the Bloom filter during operation. Adjust the file element threshold in the Bloom filter based on file deduplication information and system load information.

[0084] In some embodiments, the file deduplication information includes the actual false positive rate of file elements and the probability of file elements being deduplicated. Additionally, the system load information during the operation of the Bloom filter mentioned above can refer to the load level of the Bloom filter during operation. This system load information can be, for example, the load of the Central Processing Unit (CPU), memory, etc., without specific limitations here.

[0085] Specifically, by monitoring a second metric of the Bloom filter, which may include file deduplication information of the Bloom filter and system load information during the operation of the Bloom filter, the threshold of file elements in the Bloom filter can be adjusted based on the file deduplication information and system load information.

[0086] In this embodiment, the element threshold of the Bloom filter can be dynamically adjusted by monitoring file deduplication information and system load information. While ensuring a controllable false positive rate, the system can adaptively adjust resource usage according to the actual scenario: under high load, high frequency of access, or increased false positive rate, the element threshold is appropriately increased to improve processing performance; under low load, low frequency of access, or low false positive rate, the element threshold is decreased to conserve system resources. This method achieves an effective balance between resource consumption and processing efficiency while controlling the false positive rate.

[0087] To accurately adjust the file element threshold of the Bloom filter, in one embodiment, the file element threshold in the Bloom filter is adjusted based on file deduplication information and system load information, including: Based on file deduplication information and system load information, the adjustment range of the file element threshold is calculated; The target file element threshold is calculated based on the adjustment range of the file element threshold and the file element threshold. Adjust the file element threshold in the Bloom filter to the target file element threshold.

[0088] Specifically, it can calculate the adjustment range of the file element threshold based on file deduplication information and system load information, and can calculate the target file element threshold based on the adjustment range of the file element threshold and the file element threshold, thereby adjusting the file element threshold in the Bloom filter to the target file element threshold.

[0089] In one example, if the file deduplication information includes the actual false positive rate of file elements and the frequency of file deduplication, the file element threshold can be dynamically adjusted by combining three rules (actual false positive rate of file elements, system load, and frequency of file deduplication) to better balance system performance and resource consumption.

[0090] Specifically, the following formula can be used to calculate a comprehensive score based on the actual false positive rate of file elements, system load, and file deduplication frequency: (6) in, The score for the actual misjudgment rate of file elements. Scoring of system load, A score for the frequency of document duplication. , , For the corresponding weights.

[0091] The three rules are ordered by priority as follows: Rule 1 - Actual false positive rate is too high; Rule 2 - System load is too high; Rule 3 - File deduplication frequency is too high. Because the false positive rate directly affects system accuracy, it requires the highest priority and has the highest weight. Excessive system load may lead to performance degradation, so it is adjusted next in priority. As for the excessive file deduplication frequency, it may be a temporary phenomenon and is adjusted last, with the lowest weight.

[0092] Secondly, the formula for calculating the actual false positive rate of the above-mentioned file elements can be as follows: (7) in, This represents the actual false positive rate. The expected false positive rate threshold is used. If the actual false positive rate of a file element is lower than the expected threshold, the false positive rate score is 0.

[0093] The formula for calculating the system load score is as follows: (8) For current CPU utilization, CPU threshold, For current memory usage, This represents the memory threshold. If CPU or memory usage is below the threshold, the corresponding value is 0.

[0094] The formula for calculating the plagiarism frequency score of the above documents can be shown below: (9) in, This represents the current frequency of duplicate detection. This is the frequency threshold. If the file deduplication frequency is below the threshold, the frequency score is 0.

[0095] Thus, after calculating the comprehensive score A, the adjustment range of the element threshold can be calculated based on the comprehensive score A, as shown in the following formula (10): (10) Where F is a preset adjustment coefficient, which can be determined according to the actual situation. For example, it can be set to 0.1-0.5 to control the adjustment range.

[0096] Finally, the target file element threshold can be calculated using the following formula (11): (11) in, Threshold for target file elements, This is the file element threshold. If the adjustment magnitude is negative, the file element threshold is decreased; if the adjustment magnitude is positive, the file element threshold is increased.

[0097] In this embodiment, the adjustment range of the file element threshold can be accurately calculated through file deduplication information and system load information, thereby accurately adjusting the file element threshold and better balancing system performance and resource consumption.

[0098] Because Bloom filters can misclassify files, if the first query result corresponding to the Bloom filter indicates that the file status of the file to be downloaded is "downloaded," it is necessary to query the file elements of the file to be downloaded again in the database to make a final determination. Therefore, to reduce the frequency of database queries, in one embodiment, before querying the file elements of the file to be downloaded in the database to obtain the second query result, such as... Figure 4 As shown, the document processing method provided in this application embodiment further includes: S410 queries the file elements of the file to be downloaded in the cache and obtains the third query result.

[0099] The third query result can be used to characterize the file status of the file to be downloaded.

[0100] It's important to note that this cache can store multiple falsely identified file elements. Therefore, when querying the cache for the file element of the file to be downloaded, if the file element can be found among the multiple falsely identified elements stored in the cache, it indicates that the file element of the file to be downloaded was a falsely identified element, and the third query result indicates that the file status of the file to be downloaded is "not downloaded." Conversely, if the file element of the file to be downloaded cannot be found among the multiple falsely identified elements stored in the cache, it indicates that the file element of the file to be downloaded is not a falsely identified element, and the third query result indicates that the file status of the file to be downloaded is "downloaded."

[0101] It should also be noted that each of the above misjudged file elements can be a file element that meets certain conditions among all misjudged file elements. The condition can be a file element whose number of misjudgments is greater than a preset threshold. The preset threshold can be set in advance based on actual experience or circumstances, and is not specifically limited here.

[0102] Thus, the steps described above for querying the file elements of the file to be downloaded in the database to obtain the second query result can specifically include: S2301, if the third query result indicates that the file status of the file to be downloaded is "downloaded", then query the file elements of the file to be downloaded in the database to obtain the second query result.

[0103] Specifically, if the third query result indicates that the file status of the file to be downloaded is already downloaded, the file elements of the file to be downloaded can then be queried in the database to accurately determine whether the file to be downloaded has been downloaded, thereby obtaining the corresponding second query result.

[0104] Therefore, it should also be noted that if the third query result indicates that the file status of the file to be downloaded is not downloaded, then the file to be downloaded will be downloaded.

[0105] In this embodiment, by introducing a cache before querying the file elements of the file to be downloaded in the database, and querying the file elements of the file to be downloaded in the cache, the corresponding query results can be obtained. In this way, the frequency of database queries can be effectively reduced, thereby improving the performance of deduplication verification.

[0106] Based on this, in one embodiment, the file processing method provided in this application may further include the following steps: Retrieve cached information of cached file elements; If the cached file element information meets the preset caching conditions, delete the target misjudged file element from the cache.

[0107] It should be noted that the cache stores multiple misjudged file elements, which may include the target misjudged file element.

[0108] In addition, the aforementioned file element caching information includes at least one of the following: the number of multiple falsely identified file elements stored in the cache, and the caching duration for each falsely identified file element. Correspondingly, the preset caching conditions may include at least one of the following: the number of multiple falsely identified file elements is greater than or equal to the preset number of file elements, and the caching duration for the target falsely identified file element is greater than or equal to the preset caching duration. The preset number of file elements can be a threshold value determined based on actual circumstances, and the preset caching duration can be a threshold value determined based on actual circumstances. For example, the preset caching duration can be set to 1 minute, 30 seconds, etc., without specific limitations here.

[0109] In one example, the number of multiple falsely identified file elements stored in the cache can be obtained. If the number of these multiple falsely identified file elements is greater than or equal to a preset number of file elements, the target falsely identified file element can be deleted from the multiple falsely identified file elements. This target falsely identified file element can be selected from the multiple falsely identified file elements according to a preset eviction mechanism. The preset eviction mechanism can include a Least Recently Used (LRU) mechanism or a random mechanism, which is not specifically limited here.

[0110] In another example, it is possible to obtain the cache duration of each misjudged file element in the cache. If the cache duration of the misjudged file element is greater than the preset cache duration, the misjudged file element can be used as the target misjudged file element and can be deleted from the cache.

[0111] In this embodiment, the cached information of file elements in the cache can be monitored, and if the cached information of file elements in the cache meets preset caching conditions, the target misjudged file element in the cache can be deleted. This effectively improves the utilization efficiency and data quality of the cache, thereby optimizing the overall system performance. Based on the foregoing embodiments, in order to accurately and thoroughly describe the document processing method provided in the embodiments of this application, in one example, such as Figure 5 As shown, the document processing method provided in this application embodiment may specifically include the following steps: S501 receives a download request for a file to be downloaded.

[0112] S502, use a Bloom filter to query the file elements of the file to be downloaded and obtain the first query result. If the first query result is "yes", that is, it indicates that the file status of the file to be downloaded is "downloaded", then execute S503; otherwise, execute S506.

[0113] S503: Query the file elements of the file to be downloaded in the cache to obtain the third query result. If the third query result is "no", indicating that the file status of the file to be downloaded is already downloaded, then execute S504; otherwise, execute S506.

[0114] S504: Query the file elements of the file to be downloaded in the database to obtain the second query result. If the second query result is "yes", indicating that the file status of the file to be downloaded is "downloaded", then execute S505; otherwise, execute S506.

[0115] S505, ignore download requests for the file to be downloaded.

[0116] S506, determine whether the file to be downloaded needs to be cached. If yes, proceed to step S507; otherwise, proceed to step S508.

[0117] S507, Update the file elements stored in the cache.

[0118] S508, downloading files to be downloaded.

[0119] S509, store the file to be downloaded to the database.

[0120] S510: Should the configuration parameters of the Bloom filter be dynamically updated? If yes, proceed to S511; otherwise, proceed to S512.

[0121] S511, update the configuration parameters of the Bloom filter.

[0122] S512, End.

[0123] The specific processes shown in S501 to S512 above can be found in the aforementioned embodiments, and will not be elaborated further here.

[0124] Based on this, in one example... Figure 5 The specific process for adjusting the parameters of the Bloom filter shown can be as follows: Figure 6 As shown, the process of adjusting the configuration parameters of this Bloom filter can specifically include the following steps: S601, the primary monitoring indicator.

[0125] S602, based on the first metric, calculate the system fusion state value of the Bloom filter.

[0126] S603, determine whether the system fusion status value triggers the parameter configuration threshold; if so, proceed to S604. S604, determine whether the system fusion status value has triggered the initialization threshold. If yes, execute S605; otherwise, execute S606.

[0127] S605, Initialize the Bloom filter.

[0128] S606, determine whether the parameter adjustment strategy is the initialization strategy. If yes, execute S605; otherwise, execute S607.

[0129] S607, an optimized Bloom filter.

[0130] S608, End.

[0131] The specific processes shown in S601 to S608 above can be found in the foregoing embodiments, and will not be elaborated further here.

[0132] Based on the same inventive concept, embodiments of this application provide a document processing apparatus, specifically combined with... Figure 7 The document processing apparatus provided in the embodiments of this application will be described in detail.

[0133] Figure 7 This is a schematic diagram of the structure of a document processing apparatus provided in an embodiment of this application.

[0134] like Figure 7 As shown, the device 700 for processing this document may include: The receiving module 710 is used to request a download, which includes file elements of the file to be downloaded. The query module 720 is used to respond to a download request by using a Bloom filter to query the file elements of the file to be downloaded and obtain the first query result. The configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion status value of the Bloom filter, which represents the health status of the Bloom filter. The query module 720 is also used to query the file elements of the file to be downloaded in the database to obtain a second query result when the first query result indicates that the file status of the file to be downloaded is already downloaded. Execution module 730 is used to perform processing operations on the file to be downloaded based on the second query result.

[0135] In one embodiment, the document processing apparatus provided in this application includes: The determination module is used to determine the system fusion state value of the Bloom filter; The adjustment module is used to dynamically adjust the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter. Different parameter adjustment strategies correspond to different system fusion state values.

[0136] In one embodiment, the document processing apparatus provided in this application includes: The monitoring module is used to monitor the first indicator corresponding to the Bloom filter. The first indicator includes the number of file elements stored in the Bloom filter and the false positive rate of file elements. The calculation module is used to calculate the system fusion state value of the Bloom filter based on the number of file elements and the file element false positive rate.

[0137] In one embodiment, the document processing apparatus provided in this application includes: The acquisition module is used to acquire first weight information, which includes a first weight for measuring the importance of the number of file elements and a second weight for measuring the importance of the misjudgment rate of file elements. The weighted summation module is used to perform weighted summation on the number of file elements and the file element misclassification rate based on the first weight and the second weight, so as to obtain the system fusion state value of the Bloom filter.

[0138] In one embodiment, the document processing apparatus provided in this application includes: The acquisition module is also used to acquire the first mapping relationship between the number of elements and the score, and the second mapping relationship between the false judgment rate and the score; The determination module is also used to determine the first score corresponding to the number of file elements based on the first mapping relationship; The determination module is also used to determine the second score corresponding to the misjudgment rate of file elements based on the second mapping relationship; The weighted summation module is specifically used to perform weighted summation on the first score and the second score based on the first weight and the second weight, so as to obtain the system fusion state value of the Bloom filter.

[0139] In one embodiment, the document processing apparatus provided in this application includes: The determination module is also used to determine the parameter adjustment strategy corresponding to the system fusion status value when the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold. The determination module is also used to determine the target configuration parameters for adjusting the configuration parameters of the Bloom filter when the parameter adjustment strategy corresponding to the system fusion state value is an optimization adjustment strategy; wherein the target configuration parameters include at least one of the bit array scaling factor and the function number scaling factor; The adjustment module is used to dynamically adjust the configuration parameters of the Bloom filter to the target configuration parameters according to the parameter adjustment strategy corresponding to the system fusion status value of the Bloom filter.

[0140] In one embodiment, the document processing apparatus provided in this application includes: The building module is used to construct the target Bloom filter based on the target configuration parameters; The update module is used to update the file elements stored in the Bloom filter to the target Bloom filter; The update module is also used to update the target Bloom filter to a Bloom filter.

[0141] In one embodiment, the document processing apparatus provided in this application includes: The monitoring module is used to monitor the second metric corresponding to the Bloom filter. The second metric includes the file deduplication information of the Bloom filter and the system load information of the Bloom filter during operation. The adjustment module is used to adjust the file element threshold in the Bloom filter based on file deduplication information and system load information.

[0142] In one embodiment, the document processing apparatus provided in this application includes: The calculation module is used to calculate the adjustment range of the file element threshold based on file deduplication information and system load information; The calculation module is also used to calculate the target file element threshold based on the adjustment range of the file element threshold and the file element threshold. The adjustment module is used to adjust the file element threshold in the Bloom filter to the target file element threshold.

[0143] In one embodiment, the document processing apparatus provided in this application may further include: The query module is also used to query file elements of the file to be downloaded in the cache and obtain a third query result; The query module is specifically used to query the file elements of the file to be downloaded in the database when the third query result indicates that the file status of the file to be downloaded is already downloaded, and obtain the second query result.

[0144] In one embodiment, the document processing apparatus provided in this application may further include: The retrieval module is used to retrieve cached information about cached file elements; The deletion module is used to delete the target misjudged file element from the cache when the cached file element cache information meets the preset cache conditions; The file element caching information includes at least one of the following: the number of multiple misjudged file elements stored in the cache, and the caching duration of each misjudged file element; the preset caching conditions may include at least one of the following: the number of multiple misjudged file elements is greater than or equal to the preset number of file elements, and the caching duration of the target misjudged file element is greater than or equal to the preset caching duration.

[0145] In this embodiment, after receiving a download request for a file to be downloaded, a Bloom filter is first used to query the file elements of the file to be downloaded in the download request to determine whether the file has already been downloaded. Even if the first query result output by the Bloom filter indicates that the file has been downloaded, considering the potential for false positives in Bloom filters, the file elements of the file to be downloaded can be further queried in the database to obtain a more accurate second query result. Furthermore, the configuration parameters of the Bloom filter can be dynamically adjusted based on system fusion status values ​​that characterize the health of the Bloom filter. This effectively reduces the false positive rate of the Bloom filter and allows for verification in the database, thereby effectively preventing duplicate downloads of files.

[0146] The various modules in the document processing apparatus provided in this application embodiment can achieve... Figures 2 to 6 The method steps of any of the embodiments shown herein, and the corresponding technical effects thereof, will not be described in detail here for the sake of brevity.

[0147] Figure 8 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0148] An electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0149] Specifically, the processor 801 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0150] Memory 802 may include mass storage for data or instructions. For example, and not limitingly, memory 802 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 802 may include removable or non-removable (or fixed) media. Where appropriate, memory 802 may be internal or external to an electronic device. In a particular embodiment, memory 802 is a non-volatile solid-state memory.

[0151] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0152] The processor 801 implements any of the file processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 802.

[0153] In one example, the electronic device may also include a communication interface 803 and a bus 810. For example, Figure 8 As shown, the processor 801, memory 802, and communication interface 803 are connected through bus 810 and complete communication with each other.

[0154] The communication interface 803 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0155] Bus 810 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 810 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0156] Furthermore, in conjunction with the file processing methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the file processing method provided in this application embodiment.

[0157] This application also provides a computer program product, which includes a computer program that is executed by a processor to implement the file processing method provided in this application.

[0158] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0159] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0160] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A file processing method, characterized in that, The methods include: Receive a download request for a file to be downloaded, wherein the download request includes file elements of the file to be downloaded; In response to the download request, a Bloom filter is used to query the file elements of the file to be downloaded to obtain a first query result; wherein, the configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion status value of the Bloom filter, and the system fusion status value represents the health status of the Bloom filter; If the first query result indicates that the file status of the file to be downloaded is "downloaded", then the file elements of the file to be downloaded are queried in the database to obtain the second query result; Based on the second query result, the processing operation on the file to be downloaded is performed.

2. The method according to claim 1, characterized in that, Before querying the file elements of the file to be downloaded using a Bloom filter to obtain the first query result, the method further includes: Determine the system fusion state value of the Bloom filter; The configuration parameters of the Bloom filter are dynamically adjusted according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter. Different parameter adjustment strategies correspond to different system fusion state values.

3. The method according to claim 2, characterized in that, Determining the system fusion state value of the Bloom filter includes: Monitor the first indicator corresponding to the Bloom filter, the first indicator including the number of file elements stored in the Bloom filter and the file element false positive rate; The system fusion state value of the Bloom filter is calculated based on the number of file elements and the false positive rate of the file elements.

4. The method according to claim 3, characterized in that, The calculation of the system fusion state value of the Bloom filter based on the number of file elements and the false positive rate of the file elements includes: Obtain first weight information, which includes a first weight for measuring the importance of the number of file elements and a second weight for measuring the importance of the misjudgment rate of the file elements. Based on the first weight and the second weight, the number of file elements and the misclassification rate of the file elements are weighted and summed to obtain the system fusion state value of the Bloom filter.

5. The method according to claim 4, characterized in that, Before performing a weighted summation of the number of file elements and the false positive rate of the file elements based on the first weight and the second weight to obtain the system fusion state value of the Bloom filter, the method further includes: Obtain the first mapping relationship between the number of elements and the score, and the second mapping relationship between the false judgment rate and the score; Based on the first mapping relationship, determine the first score corresponding to the number of file elements; Based on the second mapping relationship, a second score corresponding to the misjudgment rate of the file element is determined; The step of performing a weighted summation of the number of file elements and the false positive rate of file elements based on the first weight and the second weight to obtain the system fusion state value of the Bloom filter includes: Based on the first weight and the second weight, the first score and the second score are weighted and summed to obtain the system fusion state value of the Bloom filter.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes dynamically adjusting the configuration parameters of the Bloom filter according to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter. If the system fusion status value is greater than the parameter configuration threshold but less than the initialization threshold, determine the parameter adjustment strategy corresponding to the system fusion status value. When the parameter adjustment strategy corresponding to the system fusion state value is an optimization adjustment strategy, a target configuration parameter for adjusting the configuration parameters of the Bloom filter is determined; wherein, the target configuration parameter includes at least one of the bit array scaling factor and the function number scaling factor; According to the parameter adjustment strategy corresponding to the system fusion state value of the Bloom filter, the configuration parameters of the Bloom filter are dynamically adjusted to the target configuration parameters.

7. The method according to claim 6, characterized in that, The dynamic adjustment of the configuration parameters of the Bloom filter to the target configuration parameters includes: A target Bloom filter is constructed based on the target configuration parameters. Update the file elements stored in the Bloom filter to the target Bloom filter; Update the target Bloom filter to the Bloom filter.

8. The method according to claim 1, characterized in that, The method further includes: Monitor the second metric corresponding to the Bloom filter, the second metric including the file deduplication information of the Bloom filter and the system load information of the Bloom filter during operation; Based on the file deduplication information and the system load information, the file element threshold in the Bloom filter is adjusted.

9. The method of claim 8, characterized in that, The step of adjusting the file element threshold in the Bloom filter based on the file deduplication information and the system load information includes: Based on the file deduplication information and the system load information, the adjustment range of the file element threshold is calculated; The target file element threshold is calculated based on the adjustment range of the file element threshold and the file element threshold. Adjust the file element threshold in the Bloom filter to the target file element threshold.

10. The method according to claim 1, characterized in that, Before querying the file elements of the file to be downloaded in the database to obtain the second query result, the method further includes: The file elements of the file to be downloaded are queried in the cache to obtain a third query result; The step of querying the file elements of the file to be downloaded in the database to obtain the second query result includes: If the third query result indicates that the file status of the file to be downloaded is "downloaded", then the file elements of the file to be downloaded are queried in the database to obtain the second query result.

11. The method according to claim 10, characterized in that, The method further includes: Obtain the cache information of the cached file elements; If the cached file element information meets the preset caching conditions, delete the target misjudged file element from the cache; The file element caching information includes at least one of the following: the number of multiple misjudged file elements stored in the cache, and the caching duration of each misjudged file element; the preset caching condition may include at least one of the following: the number of multiple misjudged file elements is greater than or equal to the preset number of file elements, and the caching duration of the target misjudged file element is greater than or equal to the preset caching duration.

12. A document processing apparatus, characterized in that, The device includes: A receiving module is used to request a download, wherein the download request includes file elements of the file to be downloaded; A query module is used to respond to the download request by querying the file elements of the file to be downloaded using a Bloom filter to obtain a first query result; wherein, the configuration parameters of the Bloom filter are dynamically adjusted based on the system fusion status value of the Bloom filter, and the system fusion status value represents the health status of the Bloom filter; The query module is also used to query the file elements of the file to be downloaded in the database when the first query result indicates that the file status of the file to be downloaded is downloaded, so as to obtain the second query result; The execution module is used to perform processing operations on the file to be downloaded based on the second query result.

13. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the file processing method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the file processing method as described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program, which, when executed, implements the file processing method as described in any one of claims 1-11.

Images