A distributed file verification method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of big data processing, in particular to a distributed file verification method and device, computer equipment, a storage medium and a computer program product. The distributed system comprises a user terminal and a service terminal, and the method comprises the following steps: obtaining a target file to be verified, dividing the target file into a plurality of subfiles and distributing the subfiles to a plurality of user terminals; obtaining first file digests corresponding to the subfiles and generated by the user terminals respectively, wherein the first file digests are used for describing file features of the subfiles; merging the first file digests in the form of key-value pairs to obtain an intermediate result set; in response to a verification request for the target file, generating a second file digest corresponding to the intermediate result set; and obtaining a verification result of the target file according to the second file digest. The method can improve the file verification efficiency in the distributed system.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a distributed file verification method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the continuous development of the information age, information and data are increasingly used in daily life, creating a demand for high-reliability, high-performance, and scalable storage. Distributed file systems (DFS) are a technology developed to meet this demand. A DFS is a file system that stores files across multiple computer nodes. It achieves distributed storage and management of files by dividing them into blocks or fragments and distributing these blocks across different computer nodes.

[0003] Distributed file verification is a method that utilizes distributed system technology to verify the integrity and consistency of files. It ensures that files have not been tampered with or corrupted during storage and transmission across different nodes. Distributed file verification typically uses hash functions to calculate the hash value of a file. A hash value is a fixed-length string calculated based on the file's content. Any change to the file content, whether minor or complete, will result in a significant change to its hash value. Distributed file verification can improve file security and reliability, and is particularly suitable for large-scale distributed storage and transmission systems. However, it also requires considerable computational and communication resources and places certain demands on the system's scalability and fault tolerance.

[0004] In related technologies, distributed file verification typically employs manual verification to check the version and content of files within a distributed system. Specifically, a target machine is selected from each cluster, and the file content and version on that machine are used as the basis for verification. During the verification process, a summary of the file content is used as the comparison criterion, allowing each storage unit in the cluster to be compared pairwise with the target machine to obtain the final result.

[0005] However, current distributed file verification methods have the following technical problems:

[0006] Verification is carried out by comparing the summaries pairwise, which can easily lead to low verification efficiency and poor accuracy of the verification results. Summary of the Invention

[0007] Therefore, it is necessary to provide a distributed file verification method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of file verification in a distributed system, in order to address the above-mentioned technical problems.

[0008] Firstly, this application provides a distributed file verification method. The method includes:

[0009] Obtain the target file to be verified, divide the target file into several sub-files, and distribute them to several user terminals;

[0010] Each of the user terminals generates a first file digest corresponding to the sub-file, and the first file digest is used to describe the file characteristics of the sub-file;

[0011] The first file digests are merged in key-value pairs to obtain an intermediate result set;

[0012] In response to a verification request for the target file, a second file digest corresponding to the intermediate result set is generated;

[0013] The verification result of the target file is obtained based on the second file digest.

[0014] In one embodiment, after merging the first document digests in key-value pairs to obtain an intermediate result set, the method further includes:

[0015] Obtain the load status of the service terminal, and select a target service terminal from among the service terminals to process the intermediate result set based on the load status;

[0016] The intermediate result set is allocated to the target service terminal, and the verification status of the target service terminal is changed.

[0017] In one embodiment, generating a second file digest corresponding to the intermediate result set in response to a verification request for the target file includes:

[0018] In response to the status change event of the service terminal, the verification request is generated. The status change event includes the verification status change, the change in the number of service terminals, and the rotation of the service terminals.

[0019] In one embodiment, after allocating the intermediate result set to the target service terminal and changing the verification status of the target service terminal, the method further includes:

[0020] The allocation status of the intermediate result set is changed to block access to the intermediate result set by service terminals other than the target service terminal.

[0021] In one embodiment, obtaining the target file to be verified, dividing the target file into several sub-files, and distributing them to several user terminals includes:

[0022] The target file is parsed and scanned based on the first thread to obtain several sub-files;

[0023] The first file digest is obtained by processing the sub-file using the second thread, and the first thread and the second thread are independent of each other.

[0024] In one embodiment, the step of obtaining a first file digest corresponding to the sub-file generated by the user terminal, wherein the first file digest is used to describe the file characteristics of the sub-file includes:

[0025] Obtain the configuration information of the target file, and filter the target file based on the configuration information to determine the whitelisted files in the target file;

[0026] The whitelisted files are skipped when generating the first file digest.

[0027] Secondly, this application also provides a distributed file verification device. The device includes:

[0028] The file scanning module is used to obtain the target file to be verified, divide the target file into several sub-files, and distribute them to several user terminals;

[0029] The first-level summary module is used to obtain the first file summary generated by the user terminal corresponding to the sub-file, and the first file summary is used to describe the file characteristics of the sub-file;

[0030] The summary merging module is used to merge the summaries of the first file in the form of key-value pairs to obtain an intermediate result set;

[0031] The second-level summary module is used to generate a second file summary corresponding to the intermediate result set in response to a verification request for the target file.

[0032] The file verification module is used to obtain the verification result of the target file based on the second file digest.

[0033] In one embodiment, after the summary merging module, the system further includes:

[0034] The load status module is used to obtain the load status of the service terminal and select a target service terminal for processing the intermediate result set from a plurality of service terminals based on the load status.

[0035] The intermediate result allocation module is used to allocate the intermediate result set to the target service terminal and change the verification status of the target service terminal.

[0036] In one embodiment, the second re-digest module includes:

[0037] The verification triggering module is used to generate the verification request in response to the status change event of the service terminal. The status change event includes the verification status change, the change in the number of service terminals, and the rotation of the service terminals.

[0038] In one embodiment, after the intermediate result allocation module, the system further includes:

[0039] The distributed lock module is used to change the allocation state of the intermediate result set in order to intercept access to the intermediate result set by other service terminals besides the target service terminal.

[0040] In one embodiment, the file scanning module includes:

[0041] The scanning thread module is used to perform parsing and scanning on the target file based on the first thread to obtain several sub-files;

[0042] The processing thread module is used to process the sub-file based on the second thread to obtain the first file digest, wherein the first thread and the second thread are independent of each other.

[0043] In one embodiment, the first re-digest module includes:

[0044] The file filtering module is used to obtain the configuration information of the target file, filter the target file based on the configuration information, and determine the whitelisted files in the target file.

[0045] A filter skip module is used to skip the whitelisted files when generating the first file digest.

[0046] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a distributed file verification method as described in any embodiment of the first aspect.

[0047] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a distributed file verification method as described in any embodiment of the first aspect.

[0048] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of a distributed file verification method as described in any embodiment of the first aspect.

[0049] The aforementioned distributed file verification method, apparatus, computer device, storage medium, and computer program product, derived through the technical features described herein, can achieve the following beneficial effects to address the technical problems in the background art:

[0050] In file verification within a distributed system, the target file to be verified is first divided into multiple sub-files and distributed to multiple user terminals. Within each user terminal, a map phase is performed to obtain a first file digest for each sub-file. Then, all the first file digests are merged using key-value pairs to obtain an intermediate result set. This intermediate result set is sent to the service terminal, which then converts it to persistent storage in the CFS (File System). When the file verification step is triggered, the service terminal generates a second file digest based on the intermediate result set and finally obtains the file verification result based on this second file digest. In implementation, the clustered distribution of user terminals and service terminals within the distributed system allows the target file to be processed by clustered devices. The segmentation and merging algorithm improves the efficiency of target file verification, and the division of the file into multiple sub-files helps improve the accuracy of file verification. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the application environment of a distributed file verification method in one embodiment.

[0052] Figure 2 This is a schematic diagram of the first process of a distributed file verification method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the second process of a distributed file verification method in another embodiment;

[0054] Figure 4 This is a schematic diagram of the third process of a distributed file verification method in another embodiment;

[0055] Figure 5 This is a schematic diagram of the fourth process of a distributed file verification method in another embodiment;

[0056] Figure 6 This is a schematic diagram of the fifth process of a distributed file verification method in another embodiment;

[0057] Figure 7This is a schematic diagram of the sixth process of a distributed file verification method in another embodiment;

[0058] Figure 8 This is a structural block diagram of a distributed file verification device in one embodiment;

[0059] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In related technologies, distributed file verification typically employs manual verification to check the version and content of files within a distributed system. Specifically, a target machine is selected from each cluster, and the file content and version on that machine are used as the basis for verification. During the verification process, a summary of the file content is used as the comparison criterion, allowing each storage unit in the cluster to be compared pairwise with the target machine to obtain the final result.

[0062] However, current distributed file verification methods have the following technical problems:

[0063] Verification is carried out by comparing the summaries pairwise, which can easily lead to low verification efficiency and poor accuracy of the verification results.

[0064] Based on this, the distributed file verification method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown provides a distributed system comprising user terminals 102 and service terminals 104 distributed in a cluster. User terminals 102 communicate with service terminals 104 via a network. A database stores the data that service terminals 104 need to process. The database can be integrated into the service terminal 104 or hosted on a cloud or other network server. User terminals 102 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Service terminals 104 can be implemented using independent servers or a server cluster consisting of multiple servers.

[0065] In one embodiment, such as Figure 2 As shown, a distributed file verification method is provided, which can be applied to... Figure 1Taking the service terminal in the example, the following steps are included:

[0066] Step 201: Obtain the target file to be verified, divide the target file into several sub-files and distribute them to several user terminals.

[0067] The target file can refer to a data file stored in a distributed file system, where the content or version of the file needs to be verified in the application or storage. Subfiles can refer to data in subfolders further categorized from the target file's folder; subfolders can be single-level or multi-level. Since data files have editable attributes such as modification time, these attributes may affect file sorting and text summaries. Therefore, a pre-defined judgment logic can be used to determine whether files with identical text content but different modification times are inconsistent.

[0068] For example, after obtaining the target file to be verified, the target file can be divided into several sub-files according to preset block rules and the sub-files can be distributed to several user terminals for processing.

[0069] Step 203: Obtain the first file digest generated by the user terminal corresponding to the sub-file, wherein the first file digest is used to describe the file characteristics of the sub-file.

[0070] A file digest refers to the calculation and generation of a fixed-length data digest or hash value from the content of a file. In file digest processing, the data in the file is input into a specific hash algorithm for calculation. File digests can preserve file characteristics; therefore, when two files are compared and differ, their file digests will also differ. File digest algorithms can include MD5 and SHA1, among others.

[0071] For example, the current stage can be used as the map process in the algorithm, that is, the mapping process, and the first file summary corresponding to the sub-file generated by each user terminal can be obtained respectively.

[0072] Step 205: Merge the first file digests in key-value pairs to obtain an intermediate result set.

[0073] A key-value pair is a data structure used to store and represent data items consisting of a key and its corresponding value. In a key-value pair, the key is unique and used to identify and access the corresponding value.

[0074] For example, the service terminal can obtain all the first file digests and merge them into an intermediate result set in key-value pairs. The intermediate result set can be stored in a map, denoted as digestmap. The client terminal can format the intermediate result set according to specific language rules, such as JSON format.

[0075] Step 207: In response to the verification request for the target file, generate a second file digest corresponding to the intermediate result set.

[0076] For example, after obtaining the intermediate result set, the service terminal can monitor events in the distributed system. Under the trigger of specific conditions, it can start the verification process of the target file. In the verification process, the service terminal can perform a second digest encoding process on the intermediate result set to generate a second file digest corresponding to the intermediate result set.

[0077] Step 209: Obtain the verification result of the target file based on the second file digest.

[0078] For example, the service terminal can obtain the verification result of the target file based on the acquired second file digest. Specifically, the service terminal can compare the second file digests of different files from different machines, or it can find a target machine in the cluster and use the file version of the target machine as the standard version, so that other service terminals in the cluster can compare with each other machine against the target machine and obtain the verification result in the comparison.

[0079] In the above-described distributed file verification method, by reasonably deducing the technical features in the embodiments, the beneficial effect of solving the technical problems raised in the background art can be achieved:

[0080] In file verification within a distributed system, the target file to be verified is first divided into multiple sub-files and distributed to multiple user terminals. Within each user terminal, a map phase is performed to obtain a first file digest for each sub-file. Then, all the first file digests are merged using key-value pairs to obtain an intermediate result set. This intermediate result set is sent to the service terminal, which then converts it to persistent storage in the CFS (File System). When the file verification step is triggered, the service terminal generates a second file digest based on the intermediate result set and finally obtains the file verification result based on this second file digest. In implementation, the clustered distribution of user terminals and service terminals within the distributed system allows the target file to be processed by clustered devices. The segmentation and merging algorithm improves the efficiency of target file verification, and the division of the file into multiple sub-files helps improve the accuracy of file verification.

[0081] In one embodiment, it can be as follows Figure 3 As shown, after step 205, the following steps are also included:

[0082] Step 301: Obtain the load status of the service terminal, and select a target service terminal from among the service terminals to process the intermediate result set based on the load status.

[0083] Server load status refers to the workload of a server within a specific time period, that is, the amount of tasks undertaken and the resource usage of the server. Server load status can be described and measured by multiple indicators, including CPU utilization, memory utilization, disk utilization, network traffic, and request response time.

[0084] For example, during the reduce phase, a target service terminal for processing the intermediate result set can be selected from several service terminals based on the load status of the service terminals. The selection can be based on a load balancing strategy.

[0085] Step 303: Distribute the intermediate result set to the target service terminal and change the verification status of the target service terminal.

[0086] For example, after the intermediate result set is assigned to the target service terminal, the verification status of the target service terminal can be changed. For instance, the checkstatus of the target file stored in the database can be set to 0, with a value of 0 indicating an unverified file status.

[0087] In this embodiment, the load balancing strategy helps improve the stability of servers in the distributed system, thereby improving the overall efficiency of file verification.

[0088] In one embodiment, it can be as follows Figure 4 As shown, step 207 includes:

[0089] Step 401: In response to the status change event of the service terminal, generate the verification request. The status change event includes the verification status change, the change in the number of service terminals, and the rotation of the service terminals.

[0090] The term "sudden stop" can refer to a situation where a machine or device suddenly stops working or loses power during normal operation. In the embodiments of this application, it can be used to describe a service terminal failure.

[0091] For example, when a verification request is generated, it can be associated with a status change event of a service terminal within the system. The status change event can include verification status changes, changes in the number of service terminals, and service terminal rotation events. Specifically, when a new machine is added to the cluster and a cluster machine rotation event occurs, it can be determined through the checkstatus field in the database table. If the verification status of all machines except the service terminals is empty, it means that a machine rotation event has occurred in the entire cluster, and verification needs to be triggered.

[0092] In this embodiment, the file verification process is triggered by monitoring the status of the service terminal, which helps to improve the efficiency of file verification.

[0093] In one embodiment, it can be as follows Figure 5 As shown, after step 303, the following steps are also included:

[0094] Step 501: Change the allocation status of the intermediate result set to block access to the intermediate result set by other service terminals besides the target service terminal.

[0095] Distributed locks can refer to mechanisms used to implement concurrency control in distributed systems. For example, in a distributed system, when multiple nodes access shared resources simultaneously, they can ensure data consistency and avoid race conditions.

[0096] For example, based on a distributed lock architecture, after the service terminal obtains the intermediate result set, the allocation state of the intermediate result set can be changed, thereby intercepting access to the intermediate result set by other service terminals besides the target service terminal.

[0097] In this embodiment, by setting a distributed lock, repeated access to or uploading of the same data is avoided, which helps to improve the stability of the distributed file system.

[0098] In one embodiment, it can be as follows Figure 6 As shown, step 201 may include:

[0099] Step 601: Perform parsing and scanning on the target file based on the first thread to obtain several sub-files.

[0100] For example, the target file can be parsed and scanned based on the first thread to obtain several sub-files.

[0101] Step 603: Process the sub-file based on the second thread to obtain the first file digest. The first thread and the second thread are independent of each other.

[0102] For example, sub-files can also be processed based on a second thread to obtain a first file summary. Based on an asynchronous processing architecture, the first thread can be independent of the second thread.

[0103] In this embodiment, asynchronous processing is used during the file scanning process, which helps to improve the efficiency of file scanning.

[0104] In one embodiment, it can be as follows Figure 7 As shown, step 203 includes:

[0105] Step 701: Obtain the configuration information of the target file, and filter the target file based on the configuration information to determine the whitelist files in the target file.

[0106] For example, configuration information of the target file can be obtained, and whitelisted files within the target file can be determined based on the configuration information. In this way, the service terminal can filter the target file based on the whitelisted files, thereby allowing the data of the files corresponding to the whitelisted files to skip the digest generation process.

[0107] Step 703: Skip the whitelist file when generating the first file digest.

[0108] For example, the service terminal skips the whitelist file when generating the first file digest.

[0109] In this embodiment, the configuration information of the target file allows some files to skip the verification process, which can improve the flexibility of file verification.

[0110] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0111] Based on the same inventive concept, this application also provides a distributed file verification device for implementing the distributed file verification method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more embodiments of the distributed file verification device provided below can be found in the limitations of the distributed file verification method described above, and will not be repeated here.

[0112] In one embodiment, such as Figure 8 As shown, a distributed file verification device is provided, including: a file scanning module, a first digest module, a digest merging module, a second digest module, and a file verification module, wherein:

[0113] The file scanning module is used to obtain the target file to be verified, divide the target file into several sub-files, and distribute them to several user terminals;

[0114] The first-level summary module is used to obtain the first file summary generated by the user terminal corresponding to the sub-file, and the first file summary is used to describe the file characteristics of the sub-file;

[0115] The summary merging module is used to merge the summaries of the first file in the form of key-value pairs to obtain an intermediate result set;

[0116] The second-level summary module is used to generate a second file summary corresponding to the intermediate result set in response to a verification request for the target file.

[0117] The file verification module is used to obtain the verification result of the target file based on the second file digest.

[0118] In one embodiment, after the summary merging module, the system further includes:

[0119] The load status module is used to obtain the load status of the service terminal and select a target service terminal for processing the intermediate result set from a plurality of service terminals based on the load status.

[0120] The intermediate result allocation module is used to allocate the intermediate result set to the target service terminal and change the verification status of the target service terminal.

[0121] In one embodiment, the second re-digest module includes:

[0122] The verification triggering module is used to generate the verification request in response to the status change event of the service terminal. The status change event includes the verification status change, the change in the number of service terminals, and the rotation of the service terminals.

[0123] In one embodiment, after the intermediate result allocation module, the system further includes:

[0124] The distributed lock module is used to change the allocation state of the intermediate result set in order to intercept access to the intermediate result set by other service terminals besides the target service terminal.

[0125] In one embodiment, the file scanning module includes:

[0126] The scanning thread module is used to perform parsing and scanning on the target file based on the first thread to obtain several sub-files;

[0127] The processing thread module is used to process the sub-file based on the second thread to obtain the first file digest, wherein the first thread and the second thread are independent of each other.

[0128] In one embodiment, the first re-digest module includes:

[0129] The file filtering module is used to obtain the configuration information of the target file, filter the target file based on the configuration information, and determine the whitelisted files in the target file.

[0130] A filter skip module is used to skip the whitelisted files when generating the first file digest.

[0131] The modules in the aforementioned distributed file verification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0132] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a distributed document verification method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0133] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0136] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A file verification method for a distributed system, characterized in that, The distributed system includes user terminals and service terminals, and the method includes: Obtain the target file to be verified, divide the target file into several sub-files, and distribute them to several user terminals; Each user terminal generates a first file digest corresponding to the sub-file, and the first file digest is used to describe the file characteristics of the sub-file. Several first file digests are merged in the form of key-value pairs to obtain an intermediate result set, which is formatted in JSON format; During the monitoring of events within the distributed system, in response to a verification request for the target file, a second file digest corresponding to the intermediate result set is generated, including: performing digest encoding processing on the intermediate result set to generate a second file digest corresponding to the intermediate result set; The verification result of the target file is obtained based on the second file digest; The step of merging the first document digests in key-value pairs to obtain an intermediate result set further includes: Obtain the load status of the service terminal, and select a target service terminal from among the service terminals to process the intermediate result set based on the load status; The intermediate result set is allocated to the target service terminal, and the verification status of the target service terminal is changed.

2. The method according to claim 1, characterized in that, The step of generating a second file digest corresponding to the intermediate result set in response to a verification request for the target file includes: In response to the status change event of the service terminal, the verification request is generated. The status change event includes the verification status change, the change in the number of service terminals, and the rotation of the service terminals.

3. The method according to claim 1, characterized in that, After allocating the intermediate result set to the target service terminal and changing the verification status of the target service terminal, the method further includes: The allocation status of the intermediate result set is changed to block access to the intermediate result set by service terminals other than the target service terminal.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the target file to be verified, dividing the target file into several sub-files, and distributing them to several user terminals includes: The target file is parsed and scanned based on the first thread to obtain several sub-files; The first file digest is obtained by processing the sub-file using the second thread, and the first thread and the second thread are independent of each other.

5. The method according to claim 1, characterized in that, The step of obtaining the first file digest corresponding to the sub-file generated by the user terminal, wherein the first file digest is used to describe the file characteristics of the sub-file, includes: Obtain the configuration information of the target file, and filter the target file based on the configuration information to determine the whitelisted files in the target file; The whitelisted files are skipped when generating the first file digest.

6. A file verification device for a distributed system, characterized in that, The distributed system includes user terminals and service terminals, and the device includes: The file scanning module is used to obtain the target file to be verified, divide the target file into several sub-files, and distribute them to several user terminals; The first-level summary module is used to obtain a first file summary generated by the user terminal corresponding to each sub-file, wherein the first file summary is used to describe the file characteristics of the sub-file; The summary merging module is used to merge several summaries of the first file in the form of key-value pairs to obtain an intermediate result set, wherein the intermediate result set is formatted in JSON format; The second-level digest module is used to generate a second file digest corresponding to the intermediate result set in response to a verification request for the target file during the monitoring of events within the distributed system. The module includes: performing digest encoding processing on the intermediate result set to generate a second file digest corresponding to the intermediate result set. The file verification module is used to obtain the verification result of the target file based on the second file digest. The load status module is used to obtain the load status of the service terminal and select a target service terminal for processing the intermediate result set from among several service terminals based on the load status. The intermediate result allocation module is used to allocate the intermediate result set to the target service terminal and change the verification status of the target service terminal.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images