File fuzzy copying method and system based on big data file cluster

Abstract

The invention provides a file fuzzy copying method and system based on a big data file cluster, and relates to the technical field of file copying, the file fuzzy copying method comprises the steps that file content, file names and metadata feature vectors are extracted from a to-be-matched file set, and the file content feature vectors are obtained based on deep learning model coding; afterwards, a distributed computing framework is used for parallel computing of similarity scores of the to-be-matched files and the files in the target file set, the scores are obtained through weighted computing of file content, file names and metadata feature vector similarity, and a to-be-copied file list is generated through screening according to a preset threshold value. And finally, the distributed file system dynamically allocates a replication task according to system resources, executes file replication based on a data block level parallel transmission mechanism, verifies data integrity and generates a replication task execution report. According to the method, fuzzy file copying can be efficiently and accurately carried out in the big data file cluster, the file copying efficiency and accuracy are improved, and the system resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to a file copying technology, and in particular to a file fuzzy copying method and system based on a large data file cluster. Background Art

[0002] File copying is a basic and important operation in computer systems, and is widely used in scenarios such as data backup, data migration, and data synchronization. With the explosive growth of data volume, the traditional single-machine file copying method can no longer meet the processing needs of massive data, and parallel file copying technology based on a distributed computing framework has emerged. This type of technology usually divides the file to be copied into multiple data blocks, and uses the parallel processing capabilities of the distributed computing framework to distribute the data blocks to multiple computing nodes for concurrent copying, thereby significantly improving the efficiency of file copying.

[0003] However, the existing file copy technology based on the distributed computing framework still has some defects and shortcomings: 1. Lack of semantic understanding of file content: Existing technologies usually only match and copy files based on file names or file paths, and are unable to identify files with similar content but different file names, resulting in duplicate or missed copies. This is especially inefficient and error-prone when processing large amounts of unstructured data.

[0004] 2. Single feature extraction method: Existing technologies usually only rely on simple file attributes (such as file name, file size, modification time, etc.) for file comparison, ignoring the characteristics of the file content itself, and it is difficult to distinguish files with similar content but different attributes, such as different versions of documents, slightly modified pictures, etc.

[0005] 3. Lack of refined control and management: Existing technologies usually lack refined control and management mechanisms for the file copying process. For example, it is difficult to dynamically adjust the priority and resource allocation of copying tasks according to the importance and real-time requirements of the files, and it is also difficult to effectively handle and monitor errors and exceptions in the copying process. Summary of the invention

[0006] The embodiments of the present invention provide a file fuzzy copy method and system based on a large data file cluster, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention, Provides a file fuzzy copy method based on big data file clusters, including: Acquire a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; Based on a distributed computing framework, the feature vector set of files to be matched is divided into multiple subtasks, feature vector similarity calculation is performed in parallel on multiple computing nodes of the distributed computing framework, and a similarity score between each file to be matched and a file in the target file set is calculated, wherein the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, and the similarity score is filtered according to a preset similarity threshold to generate a list of files to be copied; The list of files to be copied is submitted to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

[0008] The file content feature vector is obtained by encoding the file content through a deep learning model, including: Execute the WordPiece word segmentation algorithm on the input file content to perform subword segmentation to obtain a word unit sequence, unify the word unit sequence into a preset length by dynamic truncation or padding, perform word unit encoding and position encoding on each word unit in the word unit sequence, wherein the position encoding uses a sine function and a cosine function to encode the word unit position information, and superimpose the word unit encoding and the position encoding to obtain a word unit embedding sequence; Inputting the word-element embedding sequence into a multi-layer BERT encoder, each layer of which includes a multi-head self-attention sublayer and a feed-forward neural network sublayer, and the output of each of the multi-head self-attention sublayer and the feed-forward neural network sublayer is residually connected with the input and subjected to layer normalization processing to obtain a hidden feature representation of the file content; Perform feature fusion processing on the latent feature representation, map the latent feature representation into a query matrix, a key-value matrix and a numerical matrix respectively, calculate the outputs of multiple attention heads based on the query matrix, the key-value matrix and the numerical matrix, concatenate the outputs of the multiple attention heads and obtain a file content feature vector through linear transformation, and the file content feature vector is used to characterize the semantic features of the file content.

[0009] Based on a distributed computing framework, the feature vector set of the to-be-matched files is divided into a plurality of subtasks, and feature vector similarity calculation is performed in parallel on a plurality of computing nodes of the distributed computing framework. Calculating the similarity score between each to-be-matched file and the files in the target file set includes: Divide the to-be-matched file feature vector set and the target file feature vector set into two-dimensional blocks according to preset row block sizes and column block sizes to obtain a plurality of feature vector calculation blocks, construct a calculation dependency graph based on the feature vector calculation blocks, wherein the vertices of the calculation dependency graph represent the feature vector calculation blocks, the edges of the calculation dependency graph represent the data dependency relationship between the feature vector calculation blocks, and obtain the block size information of the feature vector calculation blocks; Calculate an inter-block load balancing metric value based on the block size information of the feature vector calculation block, wherein the inter-block load balancing metric value is a ratio of a maximum calculation block size to an average calculation block size in the feature vector calculation block, and obtain a data transmission amount between calculation nodes in the calculation dependency graph; Constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; allocating the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function, and establishing a feature vector calculation task in each of the computing nodes, wherein the feature vector calculation task includes feature vector similarity calculation of the corresponding computing block; The feature vector calculation tasks are executed in parallel on the multiple computing nodes. For the feature vector calculation task of each computing node, the feature vector in the corresponding computing block is subjected to L2 normalization processing to obtain a normalized feature vector, and the cosine similarity between the normalized feature vectors is calculated by matrix multiplication to obtain a local similarity calculation result. A multi-layer tree aggregation structure is constructed based on the computational dependency graph, wherein the leaf nodes of the multi-layer tree aggregation structure correspond to the local similarity calculation results of the multiple computational nodes, and an aggregation operation is performed on the local similarity calculation results of the multiple child nodes at each layer, wherein the aggregation operation selects the maximum similarity value between the same feature vector pairs as the aggregation result, until the aggregation of all levels is completed to obtain the final similarity calculation result.

[0010] Constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; and distributing the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function includes: Constructing a computational dependency adjacency matrix, wherein the elements of the computational dependency adjacency matrix represent the dependency relationship between computational blocks; constructing a data transmission volume matrix based on the computational dependency adjacency matrix, the block size information of the eigenvector computational block, and the task allocation status, wherein the elements of the data transmission volume matrix are the total amount of data transmission between computational blocks having a dependency relationship; Constructing a multi-objective optimization function by combining the inter-block load balancing metric value, the sum of the elements of the data transmission volume matrix and the maximum load of the computing node, wherein the multi-objective optimization function includes a load balancing weight coefficient, a communication overhead weight coefficient and a computing load weight coefficient; Generate an initial task allocation plan based on the minimum-maximum criterion. Under the conditions of satisfying task integrity constraints, node capacity constraints and dependency satisfaction constraints, select the plan with the minimum objective function value from the neighborhood solution of the current allocation plan through iterative optimization as the allocation plan for the next iteration. When the optimization iteration converges or reaches a preset number of iterations, the optimal allocation plan is sent to each computing node through the distributed scheduler. The optimal allocation plan includes the mapping relationship between computing blocks and computing nodes and the task execution time constraint.

[0011] Submit the list of files to be copied to the task scheduler of the distributed file system, the task scheduler dynamically allocates the execution node of the copy task according to the system resource status, starts the file copy process on the execution node, the file copy process performs the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copy process, stores the copied files to the target storage location, and generates a copy task execution report including: Collect resource status information of execution nodes in the distributed file system and construct a resource status vector; set a resource weight vector, in which the weight value of each dimension in the resource weight vector corresponds to the importance of each dimension of the resource in the resource status vector, and perform weighted calculation on the resource status vector and the resource weight vector to obtain a comprehensive load index of the execution node; Performing feature modeling on the file to be copied to obtain a file feature vector including file size, file access frequency, and file priority; calculating a resource requirement vector of the copy task based on the file feature vector, the resource requirement vector including resource requirements of each dimension corresponding to the resource state vector; The inverse of the comprehensive load index of the execution node, the inverse of the resource state vector and the resource demand vector, and the file priority in the file feature vector are respectively multiplied by a preset balance factor and then summed to obtain an objective function value; based on the objective function value, an optimal execution node is selected, and the optimal execution node is an execution node with the largest objective function value; Calculating an initial data block size according to the file size in the file feature vector and a preset parallelism, limiting the initial data block size to be within a preset minimum data block size and a maximum data block size to obtain an actual data block size; starting a file copy process on the optimal execution node, wherein the file copy process divides the file to be copied into a plurality of data blocks according to the actual data block size; Generating a data block level checksum including a random salt value and a timestamp for each of the data blocks, generating a file level checksum based on the data block level checksum; and performing integrity check on the data transmission process based on the data block level checksum and the file level checksum; During the file copying process, the amount of data transmitted per unit time is counted to obtain the instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than the performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain the updated parallelism, and the data block size is recalculated based on the updated parallelism; at the same time, the objective function value is recalculated based on the latest resource status information and a new optimal execution node is selected; based on the new optimal execution node, the performance statistics information, resource utilization status, integrity check results, and execution time of each stage during the transmission process are recorded to generate a task execution report.

[0012] In the process of file copying, the amount of data transmitted per unit time is counted to obtain an instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than a performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain an updated parallelism, and the data block size is recalculated based on the updated parallelism, including: During the file copying process, the number of data blocks that have been transmitted and their corresponding data volume in each unit statistical time window are counted, and the data volume is divided by the duration of the unit statistical time window to obtain the instantaneous transmission rate; the total amount of completed transmission from the start time of copying to the current time is counted, and the total amount of completed transmission is divided by the consumed transmission duration to obtain the cumulative average transmission rate; The accumulated average transmission rate is multiplied by a preset performance coefficient to obtain an ideal transmission rate, and the ideal transmission rate is corrected based on the current system resource occupancy rate to obtain a performance threshold; and the instantaneous transmission rate is compared with the performance threshold; When it is detected that the instantaneous transmission rate is lower than the performance threshold, a difference between the performance threshold and the instantaneous transmission rate is calculated, and the difference is divided by the performance threshold to obtain a performance degradation ratio; and a parallelism adjustment coefficient is calculated based on the performance degradation ratio; The preset parallelism is multiplied by the parallelism adjustment coefficient to obtain an updated parallelism; the current resource usage status of the system is obtained, and the maximum supportable parallelism is calculated based on the remaining available resources; the updated parallelism is limited to the range of the maximum parallelism to obtain the actual parallelism; The data block size is recalculated based on the actual parallelism, and the remaining size of the file to be copied is divided by the actual parallelism to obtain a new data block size; the new data block size is limited to a preset minimum data block size and a maximum data block size to obtain an actual data block size; and the data content that has not yet been transmitted is re-divided into data blocks according to the actual data block size.

[0013] The method further comprises: Collecting the system load factor and historical error rate of the distributed system, calculating the load sensitivity coefficient based on the system load factor, adding the initial retry interval to the product of the load sensitivity coefficient and the system load factor to obtain a basic retry interval; calculating the exponential backoff retry interval based on the basic retry interval, the number of retries, the error rate influencing factor and the historical error rate; Count the historical data of task state transitions, calculate the transition frequencies between states, and calculate the state transition probability based on the transition frequencies; weight the historical state transition probabilities and the state transition probabilities according to the historical weight factors to obtain an updated state transition probability matrix; The retry priority is calculated based on the current number of retries of the task, the task importance factor and the task error rate, wherein the retry priority decreases as the number of retries increases, increases as the task importance factor increases, and decreases as the task error rate increases; the resource reservation ratio is calculated based on the ratio of the number of failed tasks to the total number of tasks; Collecting data verification failure rate, adjusting the retry threshold based on the data verification failure rate, the retry threshold increasing as the data verification failure rate increases; collecting data inconsistency rate, adjusting the consistency check frequency based on the data inconsistency rate, the consistency check frequency increasing as the data inconsistency rate increases; When a data block transmission fails, the failed tasks are sorted according to the retry priority, a retry is initiated according to the exponential backoff retry interval, and computing resources are allocated within the resource reservation ratio; when a data block verification fails, whether to continue retrying is determined based on the retry threshold, and a data consistency check is performed according to the consistency check frequency.

[0014] A second aspect of an embodiment of the present invention provides a file fuzzy copy system based on a large data file cluster, comprising: The first unit is used to obtain a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; The second unit is used to divide the feature vector set of the files to be matched into multiple subtasks based on a distributed computing framework, perform feature vector similarity calculation in parallel on multiple computing nodes of the distributed computing framework, calculate the similarity score between each file to be matched and the files in the target file set, the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, filter the similarity score according to a preset similarity threshold, and generate a list of files to be copied; The third unit is used to submit the list of files to be copied to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

[0015] A third aspect of the embodiments of the present invention An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0016] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0017] The beneficial effects of this application are as follows: 1. Improve file copying efficiency: Through the distributed computing framework, file similarity is calculated in parallel, file copy operations are performed in parallel, and the parallel transmission mechanism at the data block level is used, which significantly improves the copying speed of large-scale file clusters.

[0018] 2. Enhance file matching accuracy: Use deep learning models to extract file content features, and combine file name features and file metadata features to calculate similarity, which can more accurately identify fuzzy matching files and avoid incorrect duplication.

[0019] 3. Ensure data integrity and reliability: Data integrity verification is performed during the file replication process, and the task scheduling and resource management mechanism provided by the distributed file system ensures the reliability of file replication and data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a file fuzzy copy method based on a large data file cluster according to an embodiment of the present invention; Figure 2 It is a structural diagram of a file fuzzy copy system based on a big data file cluster according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0023] Figure 1 FIG. 1 is a flow chart of a file fuzzy copy method based on a large data file cluster according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Obtain a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; S102. Based on a distributed computing framework, the feature vector set of files to be matched is divided into multiple subtasks, and feature vector similarity calculation is performed in parallel on multiple computing nodes of the distributed computing framework to calculate the similarity score between each file to be matched and the files in the target file set, wherein the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, and the similarity score is filtered according to a preset similarity threshold to generate a list of files to be copied; S103. Submit the list of files to be copied to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

[0024] In an optional implementation, the file content feature vector is obtained by encoding the file content through a deep learning model, including: Execute the WordPiece word segmentation algorithm on the input file content to perform subword segmentation to obtain a word unit sequence, unify the word unit sequence into a preset length by dynamic truncation or padding, perform word unit encoding and position encoding on each word unit in the word unit sequence, wherein the position encoding uses a sine function and a cosine function to encode the word unit position information, and superimpose the word unit encoding and the position encoding to obtain a word unit embedding sequence; Inputting the word-element embedding sequence into a multi-layer BERT encoder, each layer of which includes a multi-head self-attention sublayer and a feed-forward neural network sublayer, and the output of each of the multi-head self-attention sublayer and the feed-forward neural network sublayer is residually connected with the input and subjected to layer normalization processing to obtain a hidden feature representation of the file content; Perform feature fusion processing on the latent feature representation, map the latent feature representation into a query matrix, a key-value matrix and a numerical matrix respectively, calculate the outputs of multiple attention heads based on the query matrix, the key-value matrix and the numerical matrix, concatenate the outputs of the multiple attention heads and obtain a file content feature vector through linear transformation, and the file content feature vector is used to characterize the semantic features of the file content.

[0025] The file content feature vector extraction method is used to characterize the semantic features of the file content. The specific implementation method is as follows: First, the WordPiece word segmentation algorithm is used to segment the input file content into subwords. For example, if the input file content is "This is a good example", the word sequence obtained after WordPiece word segmentation may be "[CLS] Thisis a good example [SEP]". Here, "[CLS]" and "[SEP]" are special markers, indicating the start and end of the sequence respectively.

[0026] Next, the word sequence is dynamically truncated or padded. Assuming the preset length is 10, since the current word sequence length is less than 10, padding is required. The commonly used padding method is to add a "[PAD]" mark at the end of the sequence until the preset length is reached. The padded word sequence is "[CLS] This is a good example [SEP] [PAD] [PAD][PAD]". If the word sequence length exceeds the preset length, the excess part is truncated.

[0027] Then, each word in the word-gram sequence is encoded and positionally encoded. Word-gram encoding is to convert each word-gram into a corresponding vector representation. Assuming the vocabulary size is 10,000, each word-gram can be represented by a 10,000-dimensional one-hot vector. Position encoding is to encode the position information of the word-gram into a vector. For example, the position encoding of the first word-gram can be expressed as [sin(0), cos(0), sin(2*0), cos(2*0), ...], the position encoding of the second word-gram can be expressed as [sin(1), cos(1), sin(2*1), cos(2*1), ...], and so on. The word-gram encoding and position encoding are superimposed to obtain a word-gram embedding sequence. For example, the embedding vector of the first word-gram is the sum of the word-gram encoding vector and the position encoding vector of the first word-gram.

[0028] After that, the word-element embedding sequence is input into the multi-layer BERT encoder. Assume that the BERT encoder has 12 layers. Each layer of the BERT encoder contains a multi-head self-attention sublayer and a feed-forward neural network sublayer. In the multi-head self-attention sublayer, the word-element embedding sequence is mapped to a query matrix, a key-value matrix, and a numerical matrix, respectively. For example, for the embedding vector of the first word, the query matrix, the key-value matrix, and the numerical matrix are multiplied to obtain the corresponding query vector, key-value vector, and numerical vector. The outputs of multiple attention heads are calculated based on the query matrix, the key-value matrix, and the numerical matrix. Each attention head calculates the attention weights between all word-elements and weighted sums the numerical matrix according to the weights. The outputs of multiple attention heads are concatenated and linearly transformed to obtain the output of the multi-head self-attention. The output of the multi-head self-attention sublayer is residually connected to the input and subjected to layer normalization. The feed-forward neural network sublayer performs a nonlinear transformation on the output of the multi-head self-attention, and its output is also residually connected to the input and subjected to layer normalization. Finally, the hidden feature representation of the file content is obtained.

[0029] Finally, feature fusion processing is performed on the hidden feature representation. The hidden feature representation is mapped into a query matrix, a key value matrix, and a numerical matrix respectively. The outputs of multiple attention heads are calculated based on the query matrix, the key value matrix, and the numerical matrix. The outputs of multiple attention heads are concatenated and linearly transformed to obtain a file content feature vector. For example, the output of the last layer of the BERT encoder is used as the hidden feature representation, and a 768-dimensional feature vector is obtained after feature fusion processing. This feature vector is used to represent the semantic features of the file content.

[0030] The beneficial effects of this method can be summarized in the following three aspects: 1. Improve feature expression ability: Through WordPiece segmentation and BERT encoder, the semantic information of the file content can be better captured, thereby improving the expression ability of the feature vector. For example, compared with the traditional bag-of-words model, this method can better handle polysemous words and word order information.

[0031] 2. Enhance the generalization ability of the model: The BERT encoder uses a pre-training mechanism and is trained on large-scale text data, which can learn rich language knowledge and enhance the generalization ability of the model. This means that even with less training data, the model can achieve good results.

[0032] 3. Simplified feature engineering: This method automates the feature extraction process and eliminates the need for manual complex feature engineering, thereby reducing development costs and time. For example, there is no need to manually design feature templates or rules.

[0033] In an optional implementation, based on a distributed computing framework, the feature vector set of the to-be-matched files is divided into a plurality of subtasks, and feature vector similarity calculation is performed in parallel on a plurality of computing nodes of the distributed computing framework, and the similarity score between each to-be-matched file and a file in the target file set is calculated, including: Divide the to-be-matched file feature vector set and the target file feature vector set into two-dimensional blocks according to preset row block sizes and column block sizes to obtain a plurality of feature vector calculation blocks, construct a calculation dependency graph based on the feature vector calculation blocks, wherein the vertices of the calculation dependency graph represent the feature vector calculation blocks, the edges of the calculation dependency graph represent the data dependency relationship between the feature vector calculation blocks, and obtain the block size information of the feature vector calculation blocks; Calculate an inter-block load balancing metric value based on the block size information of the feature vector calculation block, wherein the inter-block load balancing metric value is a ratio of a maximum calculation block size to an average calculation block size in the feature vector calculation block, and obtain a data transmission amount between calculation nodes in the calculation dependency graph; Constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; allocating the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function, and establishing a feature vector calculation task in each of the computing nodes, wherein the feature vector calculation task includes feature vector similarity calculation of the corresponding computing block; The feature vector calculation tasks are executed in parallel on the multiple computing nodes. For the feature vector calculation task of each computing node, the feature vector in the corresponding computing block is subjected to L2 normalization processing to obtain a normalized feature vector, and the cosine similarity between the normalized feature vectors is calculated by matrix multiplication to obtain a local similarity calculation result. A multi-layer tree aggregation structure is constructed based on the computational dependency graph, wherein the leaf nodes of the multi-layer tree aggregation structure correspond to the local similarity calculation results of the multiple computational nodes, and an aggregation operation is performed on the local similarity calculation results of the multiple child nodes at each layer, wherein the aggregation operation selects the maximum similarity value between the same feature vector pairs as the aggregation result, until the aggregation of all levels is completed to obtain the final similarity calculation result.

[0034] A file feature vector similarity calculation method based on a distributed computing framework is used to efficiently calculate the similarity between large-scale file feature vectors. The core idea of ​​this method is to process the feature vector set of the to-be-matched file and the feature vector set of the target file in blocks, calculate the similarity in parallel under the distributed computing framework, and finally aggregate the results through a tree structure.

[0035] First, prepare the feature vector set of the files to be matched and the feature vector set of the target files. Assume that there are 10,000 files to be matched, the feature vector dimension of each file is 512, and there are 1,000 target files, the feature vector dimension of each file is also 512. The two feature vector sets are represented as matrix A and matrix B, respectively. The dimension of matrix A is 10000x512, and the dimension of matrix B is 1000x512.

[0036] Next, matrix A and matrix B are divided into two-dimensional blocks according to the preset row block size and column block size. For example, the row block size is set to 1000 and the column block size is set to 128. Then matrix A will be divided into 10 row blocks and 4 column blocks, a total of 40 eigenvector calculation blocks; matrix B will be divided into 1 row block and 4 column blocks, a total of 4 eigenvector calculation blocks.

[0037] Then, a computation dependency graph is constructed based on these feature vector computation blocks. The vertices of the graph represent feature vector computation blocks, and the edges represent data dependencies between computation blocks. Since the similarity calculation of A and B requires each row block of A to be calculated with each row block of B, each feature vector computation block of A has a dependency relationship with each feature vector computation block of B. At the same time, the size information of each feature vector computation block is recorded, for example, the size of a block of A is 1000x128, and the size of a block of B is 1000x128.

[0038] Next, calculate the inter-block load balancing metric. Calculate the size of each feature vector computation block, find the largest computation block size, and calculate the average size of all computation blocks. The inter-block load balancing metric is equal to the largest computation block size divided by the average computation block size. For example, if the largest computation block size is 1000x128 and the average computation block size is 1000x128, then the inter-block load balancing metric is 1. At the same time, obtain the amount of data transferred between the computation nodes in the computation dependency graph, which depends on the size of each computation block and the network topology of the computation nodes. Assume that the amount of data transferred is an estimated value, such as 1GB.

[0039] The task scheduling optimization objective function is constructed based on the inter-block load balancing metric and the data transmission volume. The objective function aims to minimize the computation time, which can be expressed as the weighted sum of the inter-block load balancing metric and the data transmission volume. For example, the objective function can be defined as: objective function = 0.8 * inter-block load balancing metric + 0.2 * data transmission volume. By optimizing this objective function, the allocation scheme of feature vector computation blocks to computation nodes can be obtained.

[0040] Then, the feature vector calculation blocks are distributed to multiple computing nodes of the distributed computing framework, and feature vector calculation tasks are established on each computing node. Each task is responsible for calculating the feature vector similarity of the corresponding calculation block.

[0041] On each computing node, the feature vectors in the assigned computing blocks are L2 normalized. For example, the feature vectors in a block of A and a block of B are L2 normalized respectively. Then, the cosine similarity between the normalized feature vectors is calculated using matrix multiplication to obtain the local similarity calculation result. For example, the matrix product of a normalized block of A and a normalized block of B is calculated to obtain a local similarity matrix.

[0042] Finally, a multi-layer tree aggregation structure is constructed based on the computational dependency graph. The leaf nodes correspond to the local similarity calculation results of each computational node. At each layer, the aggregation operation is performed on the local similarity calculation results of multiple child nodes. The aggregation operation selects the maximum similarity value between the same feature vector pairs as the aggregation result until all levels of aggregation are completed to obtain the final similarity calculation result. For example, suppose there are two computational nodes that respectively calculate the similarity between the first block of A and the first block of B, and the similarity between the second block of A and the first block of B. The aggregation operation compares the similarity values ​​of the same feature vector pairs in the two results and takes the larger value as the final result.

[0043] The beneficial effects of this method can be summarized in the following three aspects: 1. Improve computing efficiency: Through distributed computing and block processing, large-scale similarity calculation tasks are decomposed into multiple small tasks for parallel execution, significantly shortening the calculation time.

[0044] 2. Optimize resource utilization: Through task scheduling optimization, the load of each computing node is balanced to avoid idle or overloaded computing resources, thereby improving the utilization of computing resources.

[0045] 3. Improve calculation accuracy: Through the tree aggregation structure, the local results of each calculation node can be effectively integrated and the optimal similarity value can be selected, thereby improving the accuracy of the final calculation result.

[0046] In an optional implementation, constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; allocating the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function includes: Constructing a computational dependency adjacency matrix, wherein the elements of the computational dependency adjacency matrix represent the dependency relationship between computational blocks; constructing a data transmission volume matrix based on the computational dependency adjacency matrix, the block size information of the eigenvector computational block, and the task allocation status, wherein the elements of the data transmission volume matrix are the total amount of data transmission between computational blocks having a dependency relationship; Constructing a multi-objective optimization function by combining the inter-block load balancing metric value, the sum of the elements of the data transmission volume matrix and the maximum load of the computing node, wherein the multi-objective optimization function includes a load balancing weight coefficient, a communication overhead weight coefficient and a computing load weight coefficient; Generate an initial task allocation plan based on the minimum-maximum criterion. Under the conditions of satisfying task integrity constraints, node capacity constraints and dependency satisfaction constraints, select the plan with the minimum objective function value from the neighborhood solution of the current allocation plan through iterative optimization as the allocation plan for the next iteration. When the optimization iteration converges or reaches a preset number of iterations, the optimal allocation plan is sent to each computing node through the distributed scheduler. The optimal allocation plan includes the mapping relationship between computing blocks and computing nodes and the task execution time constraint.

[0047] A distributed task scheduling optimization method based on load balancing and data transmission volume aims to minimize computing time and improve resource utilization. The core idea of ​​this method is to comprehensively consider the dependencies between computing blocks, data transmission volume, and the load of computing nodes, and reasonably allocate computing tasks to each computing node.

[0048] First, construct the computation dependency adjacency matrix. Traverse all computation blocks. If the output of block A is the input of block B, the element value of row A and column B in the adjacency matrix is ​​1, otherwise it is 0.

[0049] Next, the data transmission volume matrix is ​​constructed based on the calculation dependency adjacency matrix, the block size information of the eigenvector calculation block, and the task allocation status. Traverse the adjacency matrix. If the element value of row A and column B is 1, it means that there is a dependency relationship between block A and block B. Then calculate the output data volume of block A, which is the amount of data that needs to be transmitted between block A and block B. Fill the data volume into row A and column B of the data transmission volume matrix.

[0050] Then, a multi-objective optimization function is constructed. The goal of this function is to minimize the load difference between blocks, minimize the total amount of data transmission, and ensure that the load of the computing node does not exceed the maximum value. The function contains three weight coefficients: load balancing weight coefficient, communication overhead weight coefficient, and computing load weight coefficient. These coefficients are used to adjust the importance of the three objectives. For example, if you pay more attention to load balancing, you can set the load balancing weight coefficient to be larger.

[0051] Subsequently, an initial task allocation scheme is generated based on the minimum-maximum criterion. The scheme needs to satisfy three constraints: task integrity constraint (all tasks must be assigned to computing nodes), node capacity constraint (the load of each computing node cannot exceed its maximum capacity), and dependency satisfaction constraint (if block A depends on block B, then block A must be executed after block B, and they need to be assigned to the same computing node or different computing nodes with network connection).

[0052] Next, the task allocation scheme is improved through iterative optimization. In each iteration, the scheme with the minimum objective function value is selected from the neighborhood solutions of the current allocation scheme as the allocation scheme for the next iteration. The neighborhood solution refers to the solution obtained by making a small change to the current solution, such as moving a computing block from one computing node to another computing node.

[0053] Finally, when the optimization iteration converges or reaches the preset number of iterations, the optimal allocation plan is sent to each computing node through the distributed scheduler. The optimal allocation plan contains the mapping relationship between computing blocks and computing nodes and the task execution time constraint. For example, the plan may specify that block 1 is allocated to node 1 with an execution time of 10 seconds; block 2 and block 3 are allocated to node 2 with execution times of 5 seconds and 5 seconds respectively; block 4 is allocated to node 1 with an execution time of 5 seconds.

[0054] The beneficial effects of this method can be summarized in the following three aspects: 1. Improved cluster resource utilization: Through load balancing, the situation where some computing nodes are overloaded while other nodes are idle is avoided, thereby making better use of cluster resources.

[0055] 2. Reduced task completion time: By optimizing the amount of data transmission, the data exchange time between computing nodes is reduced, thereby shortening the completion time of the entire task.

[0056] 3. Enhanced scalability of the system: This method can adapt to clusters of different sizes, and can adjust parameters according to actual conditions to achieve the best scheduling effect.

[0057] In an optional implementation, the list of files to be copied is submitted to a task scheduler of a distributed file system, the task scheduler dynamically allocates execution nodes of the copy task according to the system resource status, starts a file copy process on the execution node, the file copy process performs a file copy operation based on a parallel transmission mechanism at the data block level, verifies the integrity of the data during the copy process, stores the copied files in a target storage location, and generates a copy task execution report including: Collect resource status information of execution nodes in the distributed file system and construct a resource status vector; set a resource weight vector, in which the weight value of each dimension in the resource weight vector corresponds to the importance of each dimension of the resource in the resource status vector, and perform weighted calculation on the resource status vector and the resource weight vector to obtain a comprehensive load index of the execution node; Performing feature modeling on the file to be copied to obtain a file feature vector including file size, file access frequency, and file priority; calculating a resource requirement vector of the copy task based on the file feature vector, the resource requirement vector including resource requirements of each dimension corresponding to the resource state vector; The inverse of the comprehensive load index of the execution node, the inverse of the resource state vector and the resource demand vector, and the file priority in the file feature vector are respectively multiplied by a preset balance factor and then summed to obtain an objective function value; based on the objective function value, an optimal execution node is selected, and the optimal execution node is an execution node with the largest objective function value; Calculating an initial data block size according to the file size in the file feature vector and a preset parallelism, limiting the initial data block size to be within a preset minimum data block size and a maximum data block size to obtain an actual data block size; starting a file copy process on the optimal execution node, wherein the file copy process divides the file to be copied into a plurality of data blocks according to the actual data block size; Generating a data block level checksum including a random salt value and a timestamp for each of the data blocks, generating a file level checksum based on the data block level checksum; and performing integrity check on the data transmission process based on the data block level checksum and the file level checksum; During the file copying process, the amount of data transmitted per unit time is counted to obtain the instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than the performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain the updated parallelism, and the data block size is recalculated based on the updated parallelism; at the same time, the objective function value is recalculated based on the latest resource status information and a new optimal execution node is selected; based on the new optimal execution node, the performance statistics information, resource utilization status, integrity check results, and execution time of each stage during the transmission process are recorded to generate a task execution report.

[0058] First, collect the resource status information of each execution node in the distributed file system. This information includes CPU usage, memory usage, network bandwidth usage, disk I / O rate, etc. For example, the CPU usage of node A is 20%, the memory usage is 30%, the network bandwidth usage is 10%, and the disk I / O rate is 40MB / s. This information is constructed into a resource status vector, for example, the resource status vector of node A is (20, 30, 10, 40).

[0059] Then, set the resource weight vector to indicate the importance of different resource dimensions in the comprehensive load evaluation. For example, if you think that CPU and memory are more important than network bandwidth and disk I / O, you can set the resource weight vector to (0.4, 0.4, 0.1, 0.1). Multiply the corresponding dimensions of the resource state vector and the resource weight vector and sum them to get the comprehensive load index of the execution node. For example, the comprehensive load index of node A is 20 * 0.4 + 30 * 0.4 + 10 * 0.1 + 40 *0.1 = 23.

[0060] Next, perform feature modeling on the files to be copied. Extract feature information such as file size, file access frequency, and file priority to construct a file feature vector. For example, if the size of file F1 is 1GB, the access frequency is 10 times a day, and the priority is high, then the feature vector of file F1 is (1GB, 10, high).

[0061] The resource requirement vector of the replication task is calculated based on the file feature vector. Each dimension in the resource requirement vector corresponds to a dimension in the resource state vector, indicating the corresponding amount of resources required to replicate the file. For example, to replicate file F1, a CPU usage of 20%, a memory usage of 30%, a network bandwidth of 10Mbps, and a disk I / O rate of 50MB / s are required, so the resource requirement vector is (20, 30, 10, 50).

[0062] In order to select the optimal execution node, it is necessary to calculate the objective function value of each execution node. The inverse of the comprehensive load index of the execution node, the corresponding dimensions of the inverse of the resource state vector and the resource demand vector are multiplied and summed, and the file priority is multiplied by the preset balance factor and summed to obtain the objective function value. Assume that the balance factors are 0.5, 0.3, and 0.2 respectively, the inverse of the comprehensive load index of node A is 1 / 23, and the corresponding dimensions of the inverse of the resource state vector and the resource demand vector are multiplied and summed to (1 / 20*20) + (1 / 30*30) + (1 / 10*10) + (1 / 40*50) = 3.25. The file priority is high. Assuming that the corresponding value of "high" is 3, the objective function value of node A is 0.5 * (1 / 23) + 0.3 * 3.25 + 0.2 *3 = 1.59. Select the execution node with the largest objective function value as the optimal execution node.

[0063] After determining the optimal execution node, calculate the initial data block size based on the file size and the preset parallelism. For example, if the file F1 size is 1GB and the preset parallelism is 4, the initial data block size is 256MB. The initial data block size is limited to the preset minimum data block size and maximum data block size range. For example, if the minimum data block size is 64MB and the maximum data block size is 512MB, the actual data block size is 256MB.

[0064] Start the file copy process on the optimal execution node and divide the file into multiple data blocks according to the actual data block size. Generate a data block level checksum containing a random salt value and a timestamp for each data block, and generate a file level checksum based on the data block level checksum. For example, the checksum of data block 1 is "ABCD1234567890", and the file level checksum is the combination of all data block checksums "ABCD1234567890EFGH9876543210". During the data transmission process, the data block level checksum and file level checksum are used for integrity verification.

[0065] During the file copy process, the amount of data transferred per unit time is counted to obtain the instantaneous transfer rate, and the cumulative average transfer rate of the entire transfer process is calculated. If the instantaneous transfer rate is lower than the performance threshold determined based on the cumulative average transfer rate, the preset parallelism is adjusted, the data block size is recalculated, and the optimal execution node is reselected based on the latest resource status information. For example, if the instantaneous transfer rate is lower than 80% of the cumulative average transfer rate, the parallelism is adjusted from 4 to 8, the data block size is recalculated, and the optimal execution node is reselected.

[0066] Finally, a task execution report is generated to record performance statistics, resource utilization, integrity check results, execution time of each stage, and other information during the transmission process.

[0067] The beneficial effects of this method can be summarized in the following three aspects: 1. Improve file copying efficiency: Through the parallel transmission mechanism at the data block level and the dynamic adjustment of parallelism, the system resources can be fully utilized to improve the file copying efficiency.

[0068] 2. Ensure data integrity: Through data block level and file level checksum mechanism, ensure the integrity of data transmission process and avoid data damage or loss.

[0069] 3. Optimize resource utilization: Through dynamic resource allocation and load balancing strategies, replication tasks are assigned to the most appropriate execution nodes to optimize resource utilization and avoid resource waste and bottlenecks.

[0070] In an optional implementation, during the file copying process, the amount of data transmitted per unit time is counted to obtain an instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than a performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain an updated parallelism, and the data block size is recalculated based on the updated parallelism, including: During the file copying process, the number of data blocks that have been transmitted and their corresponding data volume in each unit statistical time window are counted, and the data volume is divided by the duration of the unit statistical time window to obtain the instantaneous transmission rate; the total amount of completed transmission from the start time of copying to the current time is counted, and the total amount of completed transmission is divided by the consumed transmission duration to obtain the cumulative average transmission rate; The accumulated average transmission rate is multiplied by a preset performance coefficient to obtain an ideal transmission rate, and the ideal transmission rate is corrected based on the current system resource occupancy rate to obtain a performance threshold; and the instantaneous transmission rate is compared with the performance threshold; When it is detected that the instantaneous transmission rate is lower than the performance threshold, a difference between the performance threshold and the instantaneous transmission rate is calculated, and the difference is divided by the performance threshold to obtain a performance degradation ratio; and a parallelism adjustment coefficient is calculated based on the performance degradation ratio; The preset parallelism is multiplied by the parallelism adjustment coefficient to obtain an updated parallelism; the current resource usage status of the system is obtained, and the maximum supportable parallelism is calculated based on the remaining available resources; the updated parallelism is limited to the range of the maximum parallelism to obtain the actual parallelism; The data block size is recalculated based on the actual parallelism, and the remaining size of the file to be copied is divided by the actual parallelism to obtain a new data block size; the new data block size is limited to a preset minimum data block size and a maximum data block size to obtain an actual data block size; and the data content that has not yet been transmitted is re-divided into data blocks according to the actual data block size.

[0071] First, initialize the file copy parameters. Set the preset parallelism, for example, the initial setting is 4, which means that the file is initially divided into 4 data blocks and transmitted simultaneously. Set the performance coefficient, for example, 0.8, for calculating the ideal transmission rate. Set the minimum data block size and the maximum data block size, for example, 1MB and 16MB respectively, to limit the fluctuation range of the data block size. Set the unit statistical time window, for example, 1 second, for calculating the instantaneous transmission rate.

[0072] Next, start file replication and perform real-time performance monitoring. At the end of each unit statistical time window, for example, every 1 second, count the number of data blocks that have been transmitted in the time window and their corresponding data volume. For example, in the first 1 second, 2 data blocks are transmitted, with a data volume of 2MB. Divide the data volume by the length of the unit statistical time window, that is, 2MB / 1 second = 2MB / s, to get the instantaneous transmission rate. At the same time, count the total amount of completed transmission from the start time of replication to the current time and the transmission time consumed. For example, assuming that 1 second has been consumed and the total amount of completed transmission is 2MB, the cumulative average transmission rate is 2MB / 1 second = 2MB / s.

[0073] Then, calculate the performance threshold. Multiply the cumulative average transmission rate by the preset performance coefficient to get the ideal transmission rate. For example, 2MB / s * 0.8 = 1.6MB / s. Consider the current system resource utilization and modify the ideal transmission rate to get the performance threshold. For example, assuming that the current CPU utilization is 50% and the memory utilization is 30%, according to the preset resource utilization impact rule, reduce the ideal transmission rate by 20%, and get the performance threshold of 1.6MB / s * (1 - 0.2) = 1.28MB / s.

[0074] Next, compare the instantaneous transfer rate with the performance threshold. If the instantaneous transfer rate is lower than the performance threshold, you need to adjust the parallelism and data block size. For example, suppose that in the second 1 second, the instantaneous transfer rate drops to 1MB / s, which is lower than the performance threshold of 1.28MB / s.

[0075] If the instantaneous transfer rate is lower than the performance threshold, the parallelism adjustment coefficient is calculated. The difference between the performance threshold and the instantaneous transfer rate is calculated, and the difference is divided by the performance threshold to obtain the performance degradation ratio. For example, (1.28MB / s - 1MB / s) / 1.28MB / s = 0.22, and the performance degradation ratio is 22%. According to the preset correspondence between the performance degradation ratio and the parallelism adjustment coefficient, for example, the parallelism adjustment coefficient corresponding to the performance degradation of 20%~30% is 1.2, then the preset parallelism is multiplied by the parallelism adjustment coefficient to obtain the updated parallelism. For example, 4 * 1.2 = 4.8.

[0076] Get the current resource usage status of the system, such as CPU usage, memory usage, etc., and calculate the maximum supported parallelism based on the remaining available resources. For example, assuming that the maximum parallelism allowed by the current system resources is 6, limit the updated parallelism to the range of the maximum parallelism to obtain the actual parallelism, which is 4.8 here. Since the parallelism must be an integer, it is rounded to 4.

[0077] Recalculate the data block size based on the actual parallelism. Divide the remaining size of the file to be copied by the actual parallelism to get the new data block size. For example, assuming the remaining size of the file to be copied is 100MB and the actual parallelism is 4, the new data block size is 100MB / 4 = 25MB. Limit the new data block size to the preset minimum data block size and maximum data block size to get the actual data block size. For example, since the maximum data block size is 16MB, the actual data block size is 16MB. Re-divide the data blocks for the data content that has not yet been transferred according to the actual data block size. Subsequent data transmission will be carried out according to the new data block size and parallelism. Repeat the above steps until the file copying is completed.

[0078] This application can achieve: 1. Improve replication efficiency: This method can dynamically adjust the degree of parallelism and data block size according to network conditions and system resources, avoiding the low transmission efficiency caused by fixed parameter settings, thereby improving the overall replication efficiency.

[0079] 2. Resource adaptation: This method can dynamically adjust the degree of parallelism according to the system resource usage, avoiding system freezes caused by excessive resource occupation and ensuring the normal operation of other applications.

[0080] 3. Stable and reliable: This method can adjust according to the real-time transmission rate, avoiding transmission interruption or failure caused by network fluctuations, and improving the stability and reliability of file copying.

[0081] In an optional implementation, the method further includes: Collecting the system load factor and historical error rate of the distributed system, calculating the load sensitivity coefficient based on the system load factor, adding the initial retry interval to the product of the load sensitivity coefficient and the system load factor to obtain a basic retry interval; calculating the exponential backoff retry interval based on the basic retry interval, the number of retries, the error rate influencing factor and the historical error rate; Count the historical data of task state transitions, calculate the transition frequencies between states, and calculate the state transition probability based on the transition frequencies; weight the historical state transition probabilities and the state transition probabilities according to the historical weight factors to obtain an updated state transition probability matrix; The retry priority is calculated based on the current number of retries of the task, the task importance factor and the task error rate, wherein the retry priority decreases as the number of retries increases, increases as the task importance factor increases, and decreases as the task error rate increases; the resource reservation ratio is calculated based on the ratio of the number of failed tasks to the total number of tasks; Collecting data verification failure rate, adjusting the retry threshold based on the data verification failure rate, the retry threshold increasing as the data verification failure rate increases; collecting data inconsistency rate, adjusting the consistency check frequency based on the data inconsistency rate, the consistency check frequency increasing as the data inconsistency rate increases; When a data block transmission fails, the failed tasks are sorted according to the retry priority, a retry is initiated according to the exponential backoff retry interval, and computing resources are allocated within the resource reservation ratio; when a data block verification fails, whether to continue retrying is determined based on the retry threshold, and a data consistency check is performed according to the consistency check frequency.

[0082] A distributed system data processing method and device are intended to improve data processing efficiency and reliability.

[0083] First, obtain the system load factor and historical error rate of the distributed system. The system load factor can be calculated by comprehensively calculating indicators such as CPU utilization, memory utilization, and network traffic. For example, the current CPU utilization is 70%, the memory utilization is 80%, and the network traffic is 50Mbps. The system load factor calculated based on the preset weights is 0.75. The historical error rate refers to the proportion of task execution failures in the past period of time. For example, in the past hour, there were 1,000 tasks executed, and 10 of them failed, then the historical error rate was 1%.

[0084] Next, calculate the load sensitivity factor. The load sensitivity factor is used to dynamically adjust the retry interval according to the system load. The calculation method of the load sensitivity factor can be adjusted according to the actual situation. For example, it can be calculated based on the difference between the system load factor and the preset threshold. Assuming the preset threshold is 0.8, the load sensitivity factor is 0.8 - 0.75 = 0.05. Add the initial retry interval to the product of the load sensitivity factor and the system load factor to get the basic retry interval. Assuming the initial retry interval is 1 second, the basic retry interval is 1 + 0.05 * 0.75 = 1.0375 seconds.

[0085] Then, the exponential backoff retry interval is calculated based on the basic retry interval, the number of retries, the error rate impact factor, and the historical error rate. The error rate impact factor is used to adjust the retry interval based on the historical error rate. For example, it can be calculated based on the difference between the historical error rate and the preset threshold. Assuming the preset threshold is 2%, the error rate impact factor is 2% - 1% = 1%. Assuming the current number of retries is 2, the exponential backoff retry interval is 1.0375 * (1 + 1% * 2) = 1.058 seconds.

[0086] At the same time, the historical data of task state transitions is counted, and the transition frequency between each state is calculated. For example, the task states include "to be executed", "executing", "successful" and "failed", and the number of transitions between each state in the past period of time is counted. For example, the number of times the "to be executed" state is converted to the "executing" state is 1000 times, the number of times the "executing" state is converted to the "successful" state is 900 times, and the number of times the "executing" state is converted to the "failed" state is 100 times.

[0087] The state transition probability is calculated based on the transition frequency. For example, the probability of the "pending" state transitioning to the "executing" state is 1000 / (1000+0) = 100%, the probability of the "executing" state transitioning to the "successful" state is 900 / (900+100) = 90%, and the probability of the "executing" state transitioning to the "failed" state is 100 / (900+100) = 10%. The historical state transition probability and the currently calculated state transition probability are weighted according to the historical weight factor to obtain the updated state transition probability matrix. Assuming the historical weight factor is 0.5, the probability of the updated "executing" state transitioning to the "successful" state is (0.5 * historical probability) + (0.5 * 90%).

[0088] In addition, the retry priority is calculated based on the current number of retries, the task importance factor, and the task error rate of the task. The task importance factor is used to distinguish the priorities of different tasks. For example, the weight of an important task can be set to 1.5, and the weight of an ordinary task can be set to 1. Assuming that the current number of retries is 2, the task importance factor is 1, and the task error rate is 1%, the retry priority can be calculated according to the preset rules, for example, retry priority = 1 / (2*1*1%)=50. The retry priority decreases as the number of retries increases, increases as the task importance factor increases, and decreases as the task error rate increases. The resource reservation ratio is calculated based on the ratio of the number of failed tasks to the total number of tasks. For example, if there are currently 10 failed tasks and the total number of tasks is 1000, the resource reservation ratio is 1%.

[0089] At the same time, collect the data verification failure rate. For example, in the past hour, a total of 1000 data blocks were verified, and 10 of them failed to be verified, so the data verification failure rate is 1%. Adjust the retry threshold based on the data verification failure rate. Assuming the initial retry threshold is 3, the adjusted retry threshold is 3 + 1% = 3.01. The retry threshold increases as the data verification failure rate increases. Collect the data inconsistency rate. For example, in the past hour, a total of 1000 data consistency checks were performed, and 10 of them found data inconsistencies, so the data inconsistency rate is 1%. Adjust the consistency check frequency based on the data inconsistency rate. Assuming the initial consistency check frequency is once per minute, the adjusted consistency check frequency is once every 54 seconds. The consistency check frequency increases as the data inconsistency rate increases.

[0090] When data block transmission fails, the failed tasks are sorted according to the retry priority, retries are initiated according to the exponential backoff retry interval, and computing resources are allocated within the resource reservation ratio. When data block verification fails, whether to continue retrying is determined based on the retry threshold, and data consistency verification is performed according to the consistency check frequency.

[0091] This application can achieve: 1. Improve data processing efficiency: Dynamically adjust the retry interval according to the system load to avoid frequent retries when the system is under high load, reduce the system burden, and improve data processing efficiency. Rationally allocate computing resources to avoid resource waste and further improve data processing efficiency.

[0092] 2. Enhance data processing reliability: Predict task execution results based on task state transition probability, take measures in advance to avoid task failure. Prioritize important tasks to ensure stable operation of key businesses. Data verification and consistency check mechanisms effectively ensure data quality and enhance data processing reliability.

[0093] 3. The system is more flexible: The retry threshold and consistency check frequency are dynamically adjusted according to the data verification failure rate and data inconsistency rate, so that the system can adapt to different data quality conditions and improve the flexibility and robustness of the system.

[0094] Figure 2 FIG. 1 is a schematic diagram of the structure of a file fuzzy copy system based on a large data file cluster according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to obtain a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; The second unit is used to divide the feature vector set of the files to be matched into multiple subtasks based on a distributed computing framework, perform feature vector similarity calculation in parallel on multiple computing nodes of the distributed computing framework, calculate the similarity score between each file to be matched and the files in the target file set, the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, filter the similarity score according to a preset similarity threshold, and generate a list of files to be copied; The third unit is used to submit the list of files to be copied to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

[0095] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0096] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0097] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file fuzzy copy method based on a large data file cluster, characterized in that: include: Acquire a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; Based on a distributed computing framework, the feature vector set of files to be matched is divided into multiple subtasks, feature vector similarity calculation is performed in parallel on multiple computing nodes of the distributed computing framework, and a similarity score between each file to be matched and a file in the target file set is calculated, wherein the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, and the similarity score is filtered according to a preset similarity threshold to generate a list of files to be copied; The list of files to be copied is submitted to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

2. The method according to claim 1, characterized in that The file content feature vector is obtained by encoding the file content through a deep learning model, including: Execute the WordPiece word segmentation algorithm on the input file content to perform subword segmentation to obtain a word unit sequence, unify the word unit sequence into a preset length by dynamic truncation or padding, perform word unit encoding and position encoding on each word unit in the word unit sequence, wherein the position encoding uses a sine function and a cosine function to encode the word unit position information, and superimpose the word unit encoding and the position encoding to obtain a word unit embedding sequence; Inputting the word-element embedding sequence into a multi-layer BERT encoder, each layer of which includes a multi-head self-attention sublayer and a feed-forward neural network sublayer, and the output of each of the multi-head self-attention sublayer and the feed-forward neural network sublayer is residually connected with the input and subjected to layer normalization processing to obtain a hidden feature representation of the file content; Perform feature fusion processing on the latent feature representation, map the latent feature representation into a query matrix, a key-value matrix and a numerical matrix respectively, calculate the outputs of multiple attention heads based on the query matrix, the key-value matrix and the numerical matrix, concatenate the outputs of the multiple attention heads and obtain a file content feature vector through linear transformation, and the file content feature vector is used to characterize the semantic features of the file content.

3. The method according to claim 1, characterized in that Based on a distributed computing framework, the feature vector set of the to-be-matched files is divided into a plurality of subtasks, and feature vector similarity calculation is performed in parallel on a plurality of computing nodes of the distributed computing framework. Calculating the similarity score between each to-be-matched file and the files in the target file set includes: Divide the to-be-matched file feature vector set and the target file feature vector set into two-dimensional blocks according to preset row block sizes and column block sizes to obtain a plurality of feature vector calculation blocks, construct a calculation dependency graph based on the feature vector calculation blocks, wherein the vertices of the calculation dependency graph represent the feature vector calculation blocks, the edges of the calculation dependency graph represent the data dependency relationship between the feature vector calculation blocks, and obtain the block size information of the feature vector calculation blocks; Calculate an inter-block load balancing metric value based on the block size information of the feature vector calculation block, wherein the inter-block load balancing metric value is a ratio of a maximum calculation block size to an average calculation block size in the feature vector calculation block, and obtain a data transmission amount between calculation nodes in the calculation dependency graph; Constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; allocating the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function, and establishing a feature vector calculation task in each of the computing nodes, wherein the feature vector calculation task includes feature vector similarity calculation of the corresponding computing block; The feature vector calculation tasks are executed in parallel on the multiple computing nodes. For the feature vector calculation task of each computing node, the feature vector in the corresponding computing block is subjected to L2 normalization processing to obtain a normalized feature vector, and the cosine similarity between the normalized feature vectors is calculated by matrix multiplication to obtain a local similarity calculation result. A multi-layer tree aggregation structure is constructed based on the computational dependency graph, wherein the leaf nodes of the multi-layer tree aggregation structure correspond to the local similarity calculation results of the multiple computational nodes, and an aggregation operation is performed on the local similarity calculation results of the multiple child nodes at each layer, wherein the aggregation operation selects the maximum similarity value between the same feature vector pairs as the aggregation result, until the aggregation of all levels is completed to obtain the final similarity calculation result.

4. The method according to claim 3, characterized in that Constructing a task scheduling optimization objective function based on the inter-block load balancing metric and the data transmission volume; and distributing the feature vector calculation blocks to multiple computing nodes of a distributed computing framework according to the optimization result of the task scheduling optimization objective function includes: Constructing a computational dependency adjacency matrix, wherein the elements of the computational dependency adjacency matrix represent the dependency relationship between computational blocks; constructing a data transmission volume matrix based on the computational dependency adjacency matrix, the block size information of the eigenvector computational block, and the task allocation status, wherein the elements of the data transmission volume matrix are the total amount of data transmission between computational blocks having a dependency relationship; Constructing a multi-objective optimization function by combining the inter-block load balancing metric value, the sum of the elements of the data transmission volume matrix and the maximum load of the computing node, wherein the multi-objective optimization function includes a load balancing weight coefficient, a communication overhead weight coefficient and a computing load weight coefficient; Generate an initial task allocation plan based on the minimum-maximum criterion. Under the conditions of satisfying task integrity constraints, node capacity constraints and dependency satisfaction constraints, select the plan with the minimum objective function value from the neighborhood solution of the current allocation plan through iterative optimization as the allocation plan for the next iteration. When the optimization iteration converges or reaches a preset number of iterations, the optimal allocation plan is sent to each computing node through the distributed scheduler. The optimal allocation plan includes the mapping relationship between computing blocks and computing nodes and the task execution time constraint.

5. The method according to claim 1, characterized in that Submit the list of files to be copied to the task scheduler of the distributed file system, the task scheduler dynamically allocates the execution node of the copy task according to the system resource status, starts the file copy process on the execution node, the file copy process performs the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copy process, stores the copied files to the target storage location, and generates a copy task execution report including: Collect resource status information of execution nodes in the distributed file system and construct a resource status vector; set a resource weight vector, in which the weight value of each dimension in the resource weight vector corresponds to the importance of each dimension of the resource in the resource status vector, and perform weighted calculation on the resource status vector and the resource weight vector to obtain a comprehensive load index of the execution node; Performing feature modeling on the file to be copied to obtain a file feature vector including file size, file access frequency, and file priority; calculating a resource requirement vector of the copy task based on the file feature vector, the resource requirement vector including resource requirements of each dimension corresponding to the resource state vector; The inverse of the comprehensive load index of the execution node, the inverse of the resource state vector and the resource demand vector, and the file priority in the file feature vector are respectively multiplied by a preset balance factor and then summed to obtain an objective function value; based on the objective function value, an optimal execution node is selected, and the optimal execution node is an execution node with the largest objective function value; Calculating an initial data block size according to the file size in the file feature vector and a preset parallelism, limiting the initial data block size to be within a preset minimum data block size and a maximum data block size to obtain an actual data block size; starting a file copy process on the optimal execution node, wherein the file copy process divides the file to be copied into a plurality of data blocks according to the actual data block size; Generating a data block level checksum including a random salt value and a timestamp for each of the data blocks, generating a file level checksum based on the data block level checksum; and performing integrity check on the data transmission process based on the data block level checksum and the file level checksum; During the file copying process, the amount of data transmitted per unit time is counted to obtain the instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than the performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain the updated parallelism, and the data block size is recalculated based on the updated parallelism; at the same time, the objective function value is recalculated based on the latest resource status information and a new optimal execution node is selected; based on the new optimal execution node, the performance statistics information, resource utilization status, integrity check results, and execution time of each stage during the transmission process are recorded to generate a task execution report.

6. The method according to claim 5, characterized in that In the process of file copying, the amount of data transmitted per unit time is counted to obtain an instantaneous transmission rate, and the cumulative average transmission rate of the entire transmission process is calculated; when the instantaneous transmission rate is lower than a performance threshold determined based on the cumulative average transmission rate, the preset parallelism is adjusted to obtain an updated parallelism, and the data block size is recalculated based on the updated parallelism, including: During the file copying process, the number of data blocks that have been transmitted and their corresponding data volume in each unit statistical time window are counted, and the data volume is divided by the duration of the unit statistical time window to obtain the instantaneous transmission rate; the total amount of completed transmission from the start time of copying to the current time is counted, and the total amount of completed transmission is divided by the consumed transmission duration to obtain the cumulative average transmission rate; The accumulated average transmission rate is multiplied by a preset performance coefficient to obtain an ideal transmission rate, and the ideal transmission rate is corrected based on the current system resource occupancy rate to obtain a performance threshold; and the instantaneous transmission rate is compared with the performance threshold; When it is detected that the instantaneous transmission rate is lower than the performance threshold, a difference between the performance threshold and the instantaneous transmission rate is calculated, and the difference is divided by the performance threshold to obtain a performance degradation ratio; and a parallelism adjustment coefficient is calculated based on the performance degradation ratio; The preset parallelism is multiplied by the parallelism adjustment coefficient to obtain an updated parallelism; the current resource usage status of the system is obtained, and the maximum supportable parallelism is calculated based on the remaining available resources; the updated parallelism is limited to the range of the maximum parallelism to obtain the actual parallelism; The data block size is recalculated based on the actual parallelism, and the remaining size of the file to be copied is divided by the actual parallelism to obtain a new data block size; the new data block size is limited to a preset minimum data block size and a maximum data block size to obtain an actual data block size; and the data content that has not yet been transmitted is re-divided into data blocks according to the actual data block size.

7. The method according to claim 1, characterized in that The method further comprises: Collecting the system load factor and historical error rate of the distributed system, calculating the load sensitivity coefficient based on the system load factor, adding the initial retry interval to the product of the load sensitivity coefficient and the system load factor to obtain a basic retry interval; calculating the exponential backoff retry interval based on the basic retry interval, the number of retries, the error rate influencing factor and the historical error rate; Count the historical data of task state transitions, calculate the transition frequencies between states, and calculate the state transition probability based on the transition frequencies; weight the historical state transition probabilities and the state transition probabilities according to the historical weight factors to obtain an updated state transition probability matrix; The retry priority is calculated based on the current number of retries of the task, the task importance factor and the task error rate, wherein the retry priority decreases as the number of retries increases, increases as the task importance factor increases, and decreases as the task error rate increases; the resource reservation ratio is calculated based on the ratio of the number of failed tasks to the total number of tasks; Collecting data verification failure rate, adjusting the retry threshold based on the data verification failure rate, the retry threshold increasing as the data verification failure rate increases; collecting data inconsistency rate, adjusting the consistency check frequency based on the data inconsistency rate, the consistency check frequency increasing as the data inconsistency rate increases; When a data block transmission fails, the failed tasks are sorted according to the retry priority, a retry is initiated according to the exponential backoff retry interval, and computing resources are allocated within the resource reservation ratio; when a data block verification fails, whether to continue retrying is determined based on the retry threshold, and a data consistency check is performed according to the consistency check frequency.

8. A file fuzzy copy system based on a large data file cluster, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain a set of files to be matched, perform feature extraction on each file in the set of files to be matched, and construct a set of feature vectors of files to be matched, wherein the set of feature vectors of files to be matched includes a file content feature vector, a file name feature vector, and a file metadata feature vector of each file, wherein the file content feature vector is obtained by encoding the file content through a deep learning model, the file name feature vector is obtained by encoding the file name through a character-level encoding model, and the file metadata feature vector includes a numerical representation of the file size, creation time, and modification time; The second unit is used to divide the feature vector set of the files to be matched into multiple subtasks based on a distributed computing framework, perform feature vector similarity calculation in parallel on multiple computing nodes of the distributed computing framework, calculate the similarity score between each file to be matched and the files in the target file set, the similarity score is obtained by weighted calculation of the file content feature vector similarity, the file name feature vector similarity and the file metadata feature vector similarity, filter the similarity score according to a preset similarity threshold, and generate a list of files to be copied; The third unit is used to submit the list of files to be copied to the task scheduler of the distributed file system. The task scheduler dynamically allocates the execution node of the copy task according to the system resource status, and starts the file copy process on the execution node. The file copy process executes the file copy operation based on the parallel transmission mechanism at the data block level, verifies the data integrity during the copying process, stores the copied files to the target storage location, and generates a copy task execution report.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images