File uploading optimization method and device based on parallel computing

Abstract

The embodiment of the invention provides a file uploading optimization method and device based on parallel computing, and the method comprises the steps: receiving a file selected by a user, carrying out parallel file transmission operation on a data uploading module and a Hash algorithm module, receiving file data through the data uploading module, determining a corresponding data uploading result, and outputting the data uploading result. The method comprises the following steps: receiving file data through a Hash algorithm module, inputting the file data into a set Hash algorithm selection model to obtain a corresponding Hash algorithm, performing Hash repetition detection operation according to the Hash algorithm, the file data and a preset Hash value database, and determining a corresponding Hash repetition result, the data uploading result and the Hash repetition result are received in parallel through the data processing module, the corresponding uploading storage result is determined according to the preset data processing rule, the data uploading result and the Hash repetition result, and the file uploading efficiency and flexibility can be improved based on the intelligent Hash algorithm selection and parallel computing method.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a file upload optimization method and device based on parallel computing. Background Art

[0002] As the process of enterprise informatization accelerates, the number of documents increases dramatically, and document management becomes an important part of enterprise operations. However, traditional document management methods have significant drawbacks, especially the problem of uploading duplicate files is becoming increasingly prominent. This not only leads to unnecessary consumption of storage space, but also makes file retrieval, classification and management extremely complicated, increasing the operating costs and management difficulty of enterprises. Therefore, it is necessary to identify duplicate files while uploading files.

[0003] Existing solutions usually rely on file name or metadata comparison, that is, duplicate file comparison by document type, document name, and document size, but these methods cannot identify duplicate files with the same content but different file names. Furthermore, for the above problem, file hash value comparison can be used to identify files with the same content but different file names for document management.

[0004] However, the above methods usually use a preset hash algorithm and need to wait for the file upload to complete before calculating the hash value, which cannot meet the efficiency and flexibility requirements of the current document management system for duplicate file identification. There is an urgent need for a file upload optimization method that calculates the file hash value while uploading the file, thereby realizing the rapid identification and processing of duplicate files. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a file upload optimization method and device based on parallel computing, which can improve the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a file upload optimization method based on parallel computing, comprising:

[0008] The parallel processing module receives the file selected by the user, and performs parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and the preset parallel processing rules to determine the corresponding file data;

[0009] The file data is received through the data upload module, a data upload operation is performed on the file data, and a corresponding data upload result is determined; the file data is received through the hash algorithm module, the file data is input into a set hash algorithm selection model, a corresponding hash algorithm is obtained, a hash duplication detection operation is performed according to the hash algorithm, the file data and a preset hash value database, and a corresponding hash duplication result is determined, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0010] The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload to the warehouse result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload to the warehouse result includes file update storage or file termination storage.

[0011] Further, performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and the preset parallel processing rule to determine the corresponding file data includes:

[0012] Perform a parallel process start operation according to the file and preset parallel processing rules, and determine a corresponding parallel allocation path;

[0013] According to the parallel allocation path, parallel file transfer operations are performed on the data upload module and the hash algorithm module respectively to determine corresponding file data.

[0014] Furthermore, the performing of a parallel process startup operation according to the file and a preset parallel processing rule to determine a corresponding parallel allocation path includes:

[0015] Performing a data channel construction operation according to the file and a preset data access rule to determine a corresponding data access channel, wherein the preset data access rule includes at least one of shared memory, file mapping, and data stream;

[0016] A parallel process startup operation is performed according to the data access channel and the preset parallel task scheduling rule to determine the corresponding parallel allocation path.

[0017] Furthermore, before receiving the file data through the hash algorithm module and inputting the file data into a set hash algorithm selection model to obtain a corresponding hash algorithm, the method further includes:

[0018] Acquire a historical file data set in the file management system, perform a feature extraction operation on the historical file data set according to a preset feature extraction rule, and determine a corresponding feature data set;

[0019] Perform a labeling operation on the historical file data set according to a preset optimal hash tag to determine a corresponding label data set;

[0020] A training set construction operation is performed on the feature data set and the label data set, and the training set obtained through the training set construction operation is input into a preset gradient boosting tree model for model training operation, and a corresponding set hash algorithm selection model is determined.

[0021] Further, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database to determine a corresponding hash duplication result includes:

[0022] Performing a hash value calculation operation on the file data according to the hash algorithm to determine a corresponding file hash value;

[0023] A hash duplication detection operation is performed according to the file hash value and a preset hash value database to determine a corresponding hash duplication result.

[0024] Further, the corresponding upload and storage result is determined according to the preset data processing rule, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage, including:

[0025] Determine whether the hash repetition result is a repetition;

[0026] If so, the data upload module is subjected to a data termination upload instruction sending operation according to the preset synchronization mechanism, the user selection is obtained, and the corresponding upload and storage result is determined according to the user selection;

[0027] If not, the file data is received to perform a database update operation on the database to determine the corresponding upload result.

[0028] Further, the method further comprises determining a corresponding upload and storage result according to a preset data processing rule, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage, and further comprises:

[0029] Determine whether the data upload result is upload completion;

[0030] If completed, receiving the hash repetition result, and determining whether the hash repetition result is a repetition;

[0031] If yes, obtain the user's selection, and determine the corresponding upload result according to the user's selection;

[0032] If not, the file data is received to perform a database update operation on the database to determine the corresponding upload result.

[0033] In a second aspect, the present application provides a file upload optimization device based on parallel computing, comprising:

[0034] A parallel processing module is used to receive a file selected by a user, and perform parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine the corresponding file data;

[0035] A parallel computing module, comprising a data upload module and a hash algorithm module, for receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0036] The data processing module is used to receive the data upload result and the hash duplication result in parallel, and determine the corresponding upload and storage result according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the file upload optimization method based on parallel computing when executing the program.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the file upload optimization method based on parallel computing.

[0039] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the file upload optimization method based on parallel computing.

[0040] It can be seen from the above technical scheme that the present application provides a file upload optimization method and device based on parallel computing, which receives the file selected by the user, performs parallel file transfer operations on the data upload module and the hash algorithm module, receives the file data through the data upload module, determines the corresponding data upload result, receives the file data through the hash algorithm module, inputs the file data into the set hash algorithm selection model, obtains the corresponding hash algorithm, performs a hash duplication detection operation according to the hash algorithm, the file data and the preset hash value database, determines the corresponding hash duplication result, receives the data upload result and the hash duplication result in parallel through the data processing module, and determines the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is one of the flowcharts of the file upload optimization method based on parallel computing in the embodiment of the present application;

[0043] Figure 2 This is a second flow chart of a file upload optimization method based on parallel computing in an embodiment of the present application;

[0044] Figure 3 The third flowchart of the file upload optimization method based on parallel computing in the embodiment of the present application;

[0045] Figure 4 This is a fourth flow chart of a file upload optimization method based on parallel computing in an embodiment of the present application;

[0046] Figure 5 This is a fifth flow chart of the file upload optimization method based on parallel computing in an embodiment of the present application;

[0047] Figure 6 This is a sixth flow chart of the file upload optimization method based on parallel computing in an embodiment of the present application;

[0048] Figure 7 FIG7 is a flowchart of a file upload optimization method based on parallel computing in an embodiment of the present application;

[0049] Figure 8This is a structural diagram of a file upload optimization device based on parallel computing in an embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0051] Reference numerals:

[0052] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0055] Considering that the existing file upload methods usually use a preset hash algorithm and need to wait for the file upload to complete to calculate the hash value, it is impossible to meet the requirements of the current document management system for the efficiency and flexibility of duplicate file identification and upload. The present application provides a file upload optimization method and device based on parallel computing, by receiving the file selected by the user, performing parallel file transfer operations on the data upload module and the hash algorithm module, receiving the file data through the data upload module, determining the corresponding data upload result, receiving the file data through the hash algorithm module, inputting the file data into the set hash algorithm selection model, obtaining the corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data and the preset hash value database, determining the corresponding hash duplication result, receiving the data upload result and the hash duplication result in parallel through the data processing module, and determining the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0056] In order to improve the efficiency and flexibility of file uploading based on intelligent hash algorithm selection and parallel computing method, the present application provides an embodiment of a file uploading optimization method based on parallel computing, see Figure 1 The file upload optimization method based on parallel computing specifically includes the following contents:

[0057] Step S101: receiving a file selected by a user through the parallel processing module, and performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine corresponding file data;

[0058] Optionally, in the present embodiment, the purpose of this step is to enable the file to be accessed by the data upload module and the hash algorithm module at the same time through parallel processing rules, and to perform file upload and hash calculation at the same time. This can avoid the disadvantage of having to wait for the data upload to be completed before starting the hash calculation, which wastes a lot of time.

[0059] Optionally, in this embodiment, the parallel processing rules include constructing a data channel for data access and an allocation path for data allocation according to the parallel task scheduling rules.

[0060] Specifically, the data access channel is responsible for transmitting the file data of the client to the server. Its construction process first needs to analyze the file characteristics according to the task and use the data access strategy to build the data access channel.

[0061] Optionally, the data access strategy includes at least one of shared memory, file mapping, and data stream:

[0062] Shared memory: For scenarios that require efficient data exchange, shared memory can be used as a data channel. The system allocates a memory area, which is accessed by the file upload module and the hash algorithm module to reduce the overhead of data replication. It is more suitable for small data files.

[0063] File mapping: By mapping files to the memory address space, file operations can be as efficient as accessing memory. The file upload module and hash algorithm module can directly access the mapping area to speed up data processing. It is suitable for large files and can reduce memory usage by directly mapping files to the process address space.

[0064] Data stream: For large-scale data transmission, data streams can be used to process data more flexibly. By defining the direction and rate of the data stream, resource utilization and load balancing during data transmission can be optimized. This is suitable for data stream scenarios that require simultaneous reading and processing, such as dynamic data in network transmission.

[0065] It is understandable that the above data access channels can select one or any combination of them according to the file characteristics to adapt to the data characteristics and increase the versatility of the system. Once the server receives data from the data access channel, it will start the parallel process according to the preset parallel task scheduling rules. The parallel process includes two main parallel tasks: the file upload module and the hash algorithm module.

[0066] Specifically, parallel task scheduling rules are a set of guiding strategies for receiving data transmitted by data channels and managing and allocating resources to optimize task execution in a parallel computing environment.

[0067] After receiving the data transmitted by the data channel, the server starts the parallel process and dynamically adjusts the generation of parallel allocation paths according to the current system load and hardware resources, and allocates the upload tasks and hash calculation tasks to different processing units (such as CPU cores, GPUs or dedicated hardware accelerators) to achieve true concurrent processing. Parallel task scheduling rules can be dynamically adjusted based on factors such as task priority, system load, and hardware resources.

[0068] Optionally, parallel task scheduling rules include but are not limited to:

[0069] Task division: Determine how to break down large tasks into subtasks that can be processed in parallel.

[0070] Load balancing: Ensures that each processor or core has roughly equal workload, preventing some processors from being overloaded while others are idle.

[0071] Priority Assignment: Assign priorities based on the urgency or importance of tasks to determine the order in which tasks are performed.

[0072] Resource allocation: Determine how much resources should be allocated to each task, such as CPU time, memory, I / O bandwidth, etc.

[0073] Communication management: Optimize data exchange between tasks, reduce communication overhead, and improve data transmission efficiency.

[0074] Synchronization and coordination: Ensure that tasks are properly synchronized when needed, avoiding data races and consistency issues.

[0075] Error handling: Defines the handling strategy for errors that occur during task execution, such as retry, rollback, or error recovery.

[0076] Task dependencies: Identify dependencies between tasks and arrange the execution order of tasks based on these dependencies.

[0077] Dynamic Scheduling: Dynamically adjust task scheduling strategies based on the real-time status of the system to adapt to changing loads and resource availability.

[0078] Energy management: In mobile devices or energy-constrained environments, scheduling rules may need to consider energy optimization.

[0079] Deadline: For real-time systems, scheduling rules need to take task deadlines into account to ensure that critical tasks are completed on time.

[0080] Scalability: The scheduling rules should be able to adapt to systems of different sizes, from single processors to multi-processor or distributed systems.

[0081] It is understandable that the above rules can be selected and combined according to specific application scenarios and requirements to form a parallel task scheduling strategy suitable for a specific system.

[0082] Optionally, in this embodiment, an example is given:

[0083] Suppose we have a cloud storage service where users can upload files to the cloud through client software.

[0084] First, based on the analysis of file data characteristics, the data channel selects file mapping as the main data channel mechanism because it combines the efficiency of memory access and the convenience of file operations. At the same time, in order to handle flow control and error recovery during large-scale data transmission, we use the data flow mechanism as an auxiliary.

[0085] Secondly, when the user chooses to upload a file, the client maps the file to a shared memory area through file mapping to build a data access channel. After the server receives the data transmitted by the data access channel, it starts the parallel process and dynamically adjusts the generated parallel allocation path according to the current system load and hardware resources, and assigns the file upload task to a group of threads or processes dedicated to data transmission (called the "upload task group"). At the same time, the hash calculation task is assigned to another group of threads or processes dedicated to calculation and data processing (called the "hash calculation group").

[0086] Finally, the data upload module and the hash algorithm module can access the file data through their respective allocation paths. The file upload module is responsible for processing the file data received from the data access channel (file mapping area) and writing it to the temporary storage area or final storage location of the cloud storage, and updating the upload progress in real time. The hash algorithm module synchronously accesses the file data being uploaded (through mechanisms such as shared memory or data streams), applies the hash algorithm to calculate the hash value of the data block, and compares it with the hash value stored in the database to detect duplicate files.

[0087] It can be understood that by processing these two tasks in parallel, the system is able to perform file upload and hash calculation at the same time, thereby improving the efficiency and accuracy of file upload and reducing the storage of duplicate files. The server starts the parallel process after receiving data from the data access channel, which is one of the key steps to achieve this goal.

[0088] Step S102: receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0089] Optionally, in this embodiment, the purpose of this step is to simultaneously perform data upload and hash value calculation based on the uploaded file data, obtain corresponding data upload results and hash calculation results, and lay the foundation for the subsequent system to make file duplication judgments based on the results and adopt corresponding strategies.

[0090] Optionally, in this embodiment, the technical means adopted in this step is to use an intelligent hash algorithm model to select an adaptive hash algorithm in the hash calculation module to achieve hash value calculation and hash value comparison.

[0091] Optionally, in this embodiment, the data upload module receives the file data, divides the file data into blocks (if the file is large), and sends the data blocks to the temporary storage area of ​​the cloud storage according to a preset upload strategy (such as breakpoint resume, multi-threaded upload, etc.). During the upload process, the data upload module will update the upload progress in real time and record errors and exceptions during the upload process. The data upload result is represented as two states: upload completed and upload incomplete.

[0092] Optionally, in this embodiment, the hash algorithm module is in parallel with the data upload module, and the hash algorithm module also receives the file data being uploaded through the data access channel. In order to ensure data consistency, the hash algorithm module usually accesses the same file data block as the data upload module.

[0093] Optionally, in this embodiment, the hash algorithm module dynamically selects the most suitable hash algorithm (such as SHA-1, SHA-256, MD5, etc.) for the current situation according to the file type, size or system configuration. The selection process is implemented by a set hash algorithm selection model, which outputs the corresponding hash algorithm according to the input file data characteristics and preset rules.

[0094] Specifically, the setting algorithm selection model in this embodiment is a model that minimizes the prediction loss after training, and before that, it also includes a process of training the model.

[0095] During the model training process, first, export all or a specific period of historical file data from the file management system. This data should include the file's metadata (such as file name, creation time, modification time, etc.) and the file content itself. According to the preset feature extraction rules, extract features from the historical file data set to generate a feature data set. These features should be able to reflect important information about the file content or attributes for subsequent model training.

[0096] Optionally, preset feature extraction rules:

[0097] Content features: Use hash algorithms (such as MD5, SHA-256, SHA-1) to hash the file content and generate hash values ​​as content features. In addition, file size, file format, etc. can also be considered as content-related features.

[0098] Metadata features,extract the file’s metadata, such as file name, creation time, modification time,,access count, etc., as additional features.

[0099] Advanced features: For specific types of files (such as text files, image files, etc.), more advanced features are extracted, such as keyword frequency in text files, color distribution in image files, etc.

[0100] Secondly, according to the preset optimal hash tag, the historical file data set is labeled to generate a labeled data set. The label should be able to reflect the uniqueness or repetitiveness of the file content.

[0101] Optional, optimal hash tag:

[0102] Unique label: For the only file in the database, a "unique" label is assigned.

[0103] Duplicate tags: For files with the same content but different file names or metadata, a "duplicate" tag is assigned based on the preset hash value comparison rules (if the hash values ​​are the same, they are considered duplicates), and the file is specified as a duplicate of the existing file.

[0104] Again, the feature dataset and label dataset are matched and integrated to construct a training set for model training.

[0105] Specifically, the feature dataset and the label dataset are matched according to the file ID or other unique identifier to ensure that each feature vector has a corresponding label. Then, the matched data is saved as a training set file in a certain format (such as CSV, JSON, etc.).

[0106] Finally, the training set obtained through the training set construction operation is input into the preset gradient boosting tree (GBDT) model for model training to determine the final hash algorithm selection model.

[0107] Specifically, load the training set data into the GBDT model, configure the model parameters, such as learning rate, number of trees, tree depth, etc., and then execute the model training process. Optimize the model parameters through multiple iterations to achieve the best performance on the training set. Save the trained model for subsequent selection of the hash algorithm.

[0108] It is understandable that the GBDT model can form a strong learner by combining multiple weak learners (such as decision trees), which has a good performance for the classification problem of complex data.

[0109] Optionally, in this embodiment, the hash algorithm module applies the selected hash algorithm to each file data block received to generate a corresponding hash value. At the same time, the hash algorithm module compares the calculated hash value with the hash value in the preset hash value database to detect whether there are duplicate files. This database stores the hash values ​​of previously uploaded files and their corresponding file information. If a matching hash value is found in the database, it indicates that a duplicate file exists. If no matching hash value is found, it means that the currently uploaded file is unique, and the hash algorithm module records the new hash value and its corresponding file information in the database. The hash duplication results are represented as two states: duplicate and non-duplicate.

[0110] Optionally, in this embodiment, a synchronization mechanism is preset, and the processing status of the two modules needs to be synchronized. For example, when the data upload module completes uploading a data block, it notifies the hash algorithm module that it can start hashing the data block. Similarly, when the hash algorithm module completes the hash calculation of a data block and detects duplication, it notifies the data upload module that it may need to stop uploading the remaining data blocks. The two modules need to synchronize their respective processing results to the system control center. The control center will make the final processing decision (such as direct storage or termination of storage) based on these results, and update the database to reflect these changes.

[0111] It is worth noting that because hash calculation and file upload are performed in parallel, the system execution process does not specify the order in which the corresponding results are received. Instead, the system determines whether the data needs to be stored by setting logical rules.

[0112] Generally speaking, the speed of hash calculation is faster than the speed of file upload, so the system may give priority to receiving the hash calculation results and make corresponding warehousing judgments based on the hash calculation results and the file upload results. However, this also depends on the file upload speed. Based on the influence of the upload network, hardware, processor and other facilities, the file upload speed for small files may be better than the hash calculation. At this time, the system gives priority to receiving the file upload results and makes corresponding warehousing judgments based on the hash calculation results. This logical rule is reflected in step S103.

[0113] Step S103: The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload and storage result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0114] Optionally, in this embodiment, this step involves data processing rules, namely preset logic rules.

[0115] Optionally, in this embodiment, the system starts two processes, file upload and hash calculation, at the same time. These two processes run independently and do not affect each other, so their completion time is uncertain. Therefore, after the hash calculation is completed, the system will immediately query the database to see if there is an identical hash value. At the same time, the system will also monitor the progress and status of the file upload.

[0116] Specifically, preset logical rules:

[0117] a. If the hash is not repeated and the upload is complete: the system will directly store the file because the file is unique and has been fully uploaded.

[0118] b. If a duplicate hash is found but the upload has not yet completed: the system will terminate the upload because it already knows that this is a duplicate file and there is no need to continue uploading.

[0119] c. If the hash is duplicated but the upload is complete: the system will prompt the user because the file is duplicated but has been uploaded. At this point, the user can choose to associate the existing file or terminate the upload.

[0120] Optionally, the above logic rules are divided into two main lines, which can be used to obtain storage judgment based on the priority of receiving hash duplicate results or the priority of receiving file upload results:

[0121] A. Hash algorithm line (prioritizes obtaining hash repetition results):

[0122] Step 1: Check whether a data item with the same hash value already exists in the hash value database.

[0123] If it exists, go to step 2.

[0124] If not present, skip to step 4.

[0125] Step 2: Send a "stop uploading" instruction to the data uploading module and execute step 3.

[0126] Step 3: Get the user to select an option.

[0127] If the user chooses to associate, the file upload will continue and the file will be updated and stored in the database.

[0128] If the user chooses to give up, the file will be terminated from storage.

[0129] Step 4: Wait for the data upload result to be "Upload completed" and update the file into the database.

[0130] B.Data upload line:

[0131] Step 1: Consider the results of data upload and perform preliminary verification of file content (such as file type, size, integrity, etc.).

[0132] After the upload is complete, proceed to step 2.

[0133] Upload is not completed, logic ends.

[0134] Step 2: Check if the hash repetition result is a duplicate.

[0135] If repeated, proceed to step 3.

[0136] If not, proceed to step 4.

[0137] Step 3: Get the user to select an option.

[0138] If the user chooses to associate, the file will be updated and stored in the database.

[0139] If the user chooses to give up, the file will be terminated from storage.

[0140] Step 4: Update the files and save them in the database.

[0141] It can be understood that through the above two logic lines, a closed loop of file storage judgment can be realized. For large-volume data such as video data, audio data, etc., considering factors such as cloud transmission, its hash recalculation generally takes precedence over file uploading. This embodiment can perform hash calculation while uploading the file. If a repeated hash value result is obtained, the file upload is terminated. Compared with hashing after the file upload is completed, a lot of time is saved.

[0142] This example demonstrates how this embodiment optimizes file uploads through parallel computing and intelligent hashing algorithms.

[0143] From the above description, it can be seen that the file upload optimization method based on parallel computing provided in the embodiment of the present application can perform parallel file transfer operations on the data upload module and the hash algorithm module by receiving the file selected by the user, receive the file data through the data upload module, determine the corresponding data upload result, receive the file data through the hash algorithm module, input the file data into the set hash algorithm selection model, obtain the corresponding hash algorithm, perform a hash duplication detection operation according to the hash algorithm, file data and a preset hash value database, determine the corresponding hash duplication result, receive the data upload result and the hash duplication result in parallel through the data processing module, and determine the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0144] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 2 , and can also include the following:

[0145] Step S201: performing a parallel process start operation according to the file and preset parallel processing rules, and determining a corresponding parallel allocation path;

[0146] Step S202: performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the parallel allocation path to determine corresponding file data.

[0147] Optionally, in this embodiment, the preset parallel processing rules include constructing a data channel for data access and an allocation path for data allocation according to the parallel task scheduling rules.

[0148] Specifically, the data access channel is responsible for transmitting the file data of the client to the server. Its construction process first needs to analyze the file characteristics according to the task and use the data access strategy to build the data access channel.

[0149] Specifically, parallel task scheduling rules are a set of guiding strategies for receiving data transmitted by data channels and managing and allocating resources to optimize task execution in a parallel computing environment.

[0150] After receiving the data transmitted by the data channel, the server starts the parallel process and dynamically adjusts the generation of parallel allocation paths according to the current system load and hardware resources, and allocates the upload tasks and hash calculation tasks to different processing units (such as CPU cores, GPUs or dedicated hardware accelerators) to achieve true concurrent processing. Parallel task scheduling rules can be dynamically adjusted based on factors such as task priority, system load, and hardware resources.

[0151] It can be understood that by processing these two tasks in parallel, the system is able to perform file upload and hash calculation at the same time, thereby improving the efficiency and accuracy of file upload and reducing the storage of duplicate files. The server starts the parallel process after receiving data from the data access channel, which is one of the key steps to achieve this goal.

[0152] Through step S202, this embodiment constructs a data distribution path through parallel rules, so that the data upload module and the hash calculation module can access the file data at the same time, laying a foundation for subsequent parallel calculations.

[0153] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 3 , and can also include the following:

[0154] Step S301: performing a data channel construction operation according to the file and a preset data access rule to determine a corresponding data access channel, wherein the preset data access rule includes at least one of shared memory, file mapping, and data stream;

[0155] Step S302: performing a parallel process startup operation according to the data access channel and preset parallel task scheduling rules, and determining a corresponding parallel allocation path.

[0156] Optionally, the data access strategy includes at least one of shared memory, file mapping, and data stream:

[0157] Shared memory: For scenarios that require efficient data exchange, shared memory can be used as a data channel. The system allocates a memory area, which is accessed by the file upload module and the hash algorithm module to reduce the overhead of data replication. It is more suitable for small data files.

[0158] File mapping: By mapping files to the memory address space, file operations can be as efficient as accessing memory. The file upload module and hash algorithm module can directly access the mapping area to speed up data processing. It is suitable for large files and can reduce memory usage by directly mapping files to the process address space.

[0159] Data stream: For large-scale data transmission, data streams can be used to process data more flexibly. By defining the direction and rate of the data stream, resource utilization and load balancing during data transmission can be optimized. This is suitable for data stream scenarios that require simultaneous reading and processing, such as dynamic data in network transmission.

[0160] It is understandable that the above data access channels can select one or any combination of them according to the file characteristics to adapt to the data characteristics and increase the versatility of the system. Once the server receives data from the data access channel, it will start the parallel process according to the preset parallel task scheduling rules. The parallel process includes two main parallel tasks: the file upload module and the hash algorithm module.

[0161] Optionally, parallel task scheduling rules include but are not limited to:

[0162] Task division: Determine how to break down large tasks into subtasks that can be processed in parallel.

[0163] Load balancing: Ensures that each processor or core has roughly equal workload, preventing some processors from being overloaded while others are idle.

[0164] Priority Assignment: Assign priorities based on the urgency or importance of tasks to determine the order in which tasks are performed.

[0165] Resource allocation: Determine how much resources should be allocated to each task, such as CPU time, memory, I / O bandwidth, etc.

[0166] Communication management: Optimize data exchange between tasks, reduce communication overhead, and improve data transmission efficiency.

[0167] Synchronization and coordination: Ensure that tasks are properly synchronized when needed, avoiding data races and consistency issues.

[0168] Error handling: Defines the handling strategy for errors that occur during task execution, such as retry, rollback, or error recovery.

[0169] Task dependencies: Identify dependencies between tasks and arrange the execution order of tasks based on these dependencies.

[0170] Dynamic Scheduling: Dynamically adjust task scheduling strategies based on the real-time status of the system to adapt to changing loads and resource availability.

[0171] Energy management: In mobile devices or energy-constrained environments, scheduling rules may need to consider energy optimization.

[0172] Deadline: For real-time systems, scheduling rules need to take task deadlines into account to ensure that critical tasks are completed on time.

[0173] Scalability: The scheduling rules should be able to adapt to systems of different sizes, from single processors to multi-processor or distributed systems.

[0174] It is understandable that the above rules can be selected and combined according to specific application scenarios and requirements to form a parallel task scheduling strategy suitable for a specific system.

[0175] Through step S302, this embodiment constructs a data distribution path through parallel rules, so that the data upload module and the hash calculation module can access the file data at the same time, laying a foundation for subsequent parallel calculations.

[0176] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 4 , and can also include the following:

[0177] Step S401: obtaining a historical file data set in a file management system, performing a feature extraction operation on the historical file data set according to a preset feature extraction rule, and determining a corresponding feature data set;

[0178] Step S402: performing a labeling operation on the historical file data set according to a preset optimal hash tag to determine a corresponding label data set;

[0179] Step S403: performing a training set construction operation on the feature data set and the label data set, inputting the training set obtained through the training set construction operation into a preset gradient boosting tree model for model training operation, and determining a corresponding set hash algorithm selection model.

[0180] Optionally, this step describes the training process of setting the hash algorithm selection model.

[0181] During the model training process, first, export all or a specific period of historical file data from the file management system. This data should include the file's metadata (such as file name, creation time, modification time, etc.) and the file content itself. According to the preset feature extraction rules, extract features from the historical file data set to generate a feature data set. These features should be able to reflect important information about the file content or attributes for subsequent model training.

[0182] Optionally, preset feature extraction rules:

[0183] Content features: Use hash algorithms (such as MD5, SHA-256, SHA-1) to hash the file content and generate hash values ​​as content features. In addition, file size, file format, etc. can also be considered as content-related features.

[0184] Metadata features,extract the file’s metadata, such as file name, creation time, modification time,,access count, etc., as additional features.

[0185] Advanced features: For specific types of files (such as text files, image files, etc.), more advanced features are extracted, such as keyword frequency in text files, color distribution in image files, etc.

[0186] Secondly, according to the preset optimal hash tag, the historical file data set is labeled to generate a labeled data set. The label should be able to reflect the uniqueness or repetitiveness of the file content.

[0187] Optional, optimal hash tag:

[0188] Unique label: For the only file in the database, a "unique" label is assigned.

[0189] Duplicate tags: For files with the same content but different file names or metadata, a "duplicate" tag is assigned based on the preset hash value comparison rules (if the hash values ​​are the same, they are considered duplicates), and the file is specified as a duplicate of the existing file.

[0190] Again, the feature dataset and label dataset are matched and integrated to construct a training set for model training.

[0191] Specifically, the feature dataset and the label dataset are matched according to the file ID or other unique identifier to ensure that each feature vector has a corresponding label. Then, the matched data is saved as a training set file in a certain format (such as CSV, JSON, etc.).

[0192] Finally, the training set obtained through the training set construction operation is input into the preset gradient boosting tree (GBDT) model for model training to determine the final hash algorithm selection model.

[0193] Specifically, load the training set data into the GBDT model, configure the model parameters, such as learning rate, number of trees, tree depth, etc., and then execute the model training process. Optimize the model parameters through multiple iterations to achieve the best performance on the training set. Save the trained model for subsequent selection of the hash algorithm.

[0194] It is understandable that the GBDT model can form a strong learner by combining multiple weak learners (such as decision trees), which has a good performance for the classification problem of complex data.

[0195] Through step S403, this embodiment successfully obtains a set hash algorithm selection model that can intelligently select a corresponding hash algorithm according to file characteristics, laying a foundation for subsequently improving the adaptability and speed of hash calculations.

[0196] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 5 , and can also include the following:

[0197] Step S501: performing a hash value calculation operation on the file data according to the hash algorithm to determine a corresponding file hash value;

[0198] Step S502: performing a hash duplication detection operation according to the file hash value and a preset hash value database to determine a corresponding hash duplication result.

[0199] Optionally, in this embodiment, the hash algorithm module applies the selected hash algorithm to each file data block received to generate a corresponding hash value. At the same time, the hash algorithm module compares the calculated hash value with the hash value in the preset hash value database to detect whether there are duplicate files. This database stores the hash values ​​of previously uploaded files and their corresponding file information. If a matching hash value is found in the database, it indicates that a duplicate file exists. If no matching hash value is found, it means that the currently uploaded file is unique, and the hash algorithm module records the new hash value and its corresponding file information in the database. The hash duplication results are represented as two states: duplicate and non-duplicate.

[0200] Through step S502, this embodiment successfully obtains the hash value duplication result by comparing the hash value with the database hash value, laying a foundation for subsequent storage judgment according to logical rules.

[0201] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 6 , and can also include the following:

[0202] Step S601: Determine whether the hash repetition result is a repetition;

[0203] Step S602: If yes, then according to the preset synchronization mechanism, a data termination upload instruction is sent to the data upload module, a user selection is obtained, and a corresponding upload and storage result is determined according to the user selection;

[0204] Step S603: If not, then the file data is received to perform a database update operation on the database to determine the corresponding upload result.

[0205] Optional, prioritize the hash algorithm line to get the hash duplicate result:

[0206] Step 1: Check whether a data item with the same hash value already exists in the hash value database.

[0207] If it exists, go to step 2.

[0208] If not present, skip to step 4.

[0209] Step 2: Send a "stop uploading" instruction to the data uploading module and execute step 3.

[0210] Step 3: Get the user to select an option.

[0211] If the user chooses to associate, the file upload will continue and the file will be updated and stored in the database.

[0212] If the user chooses to give up, the file will be terminated from storage.

[0213] Step 4: Wait for the data upload result to be "Upload completed" and update the file into the database.

[0214] Through step S603, this embodiment successfully determines whether the file is stored in the database based on the hash duplication result.

[0215] In one embodiment of the file upload optimization method based on parallel computing of the present application, see Figure 7 , and can also include the following:

[0216] Step S701: Determine whether the data upload result is completed;

[0217] Step S702: If completed, receiving the hash repetition result, and determining whether the hash repetition result is a repetition;

[0218] Step S703: If yes, obtain the user selection, and determine the corresponding upload result according to the user selection;

[0219] Step S704: If not, then the file data is received to perform a database update operation on the database to determine the corresponding upload result.

[0220] Optional, data upload line to get data upload results first:

[0221] Step 1: Consider the results of data upload and perform preliminary verification of file content (such as file type, size, integrity, etc.).

[0222] After the upload is complete, proceed to step 2.

[0223] Upload is not completed, logic ends.

[0224] Step 2: Check if the hash repetition result is a duplicate.

[0225] If repeated, proceed to step 3.

[0226] If not, proceed to step 4.

[0227] Step 3: Get the user to select an option.

[0228] If the user chooses to associate, the file will be updated and stored in the database.

[0229] If the user chooses to give up, the file will be terminated from storage.

[0230] Step 4: Update the files and save them in the database.

[0231] Through step S704, this embodiment successfully performs file storage determination based on the data upload result.

[0232] In order to improve the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing method, the present application provides an embodiment of a file upload optimization device based on parallel computing for implementing all or part of the contents of the file upload optimization method based on parallel computing, see Figure 8 The file upload optimization device based on parallel computing specifically includes the following contents:

[0233] The parallel processing module 10 is used to receive the file selected by the user, and perform parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and the preset parallel processing rules to determine the corresponding file data;

[0234] The parallel computing module 20 includes a data upload module and a hash algorithm module, which is used to receive the file data through the data upload module, perform a data upload operation on the file data, and determine a corresponding data upload result; receive the file data through the hash algorithm module, input the file data into a set hash algorithm selection model, obtain a corresponding hash algorithm, perform a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determine a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0235] The data processing module 30 is used to receive the data upload result and the hash duplication result in parallel, and determine the corresponding upload and storage result according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0236] From the above description, it can be seen that the file upload optimization device based on parallel computing provided in the embodiment of the present application can perform parallel file transfer operations on the data upload module and the hash algorithm module by receiving the file selected by the user, receive the file data through the data upload module, determine the corresponding data upload result, receive the file data through the hash algorithm module, input the file data into the set hash algorithm selection model, obtain the corresponding hash algorithm, perform a hash duplication detection operation according to the hash algorithm, file data and a preset hash value database, determine the corresponding hash duplication result, receive the data upload result and the hash duplication result in parallel through the data processing module, and determine the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0237] From the hardware level, in order to improve the efficiency and flexibility of file uploading based on intelligent hash algorithm selection and parallel computing methods, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the file upload optimization method based on parallel computing, and the electronic device specifically includes the following contents:

[0238] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the file upload optimization method based on parallel computing and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the file upload optimization method based on parallel computing and the embodiment of the file upload optimization method based on parallel computing in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0239] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0240] In practical applications, part of the file upload optimization method based on parallel computing can be executed on the electronic device side as described above, or all operations can be completed in the client device. The selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0241] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0242] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0243] In one embodiment, the file upload optimization method function based on parallel computing can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0244] Step S101: receiving a file selected by a user through the parallel processing module, and performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine corresponding file data;

[0245] Step S102: receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0246] Step S103: The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload and storage result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0247] From the above description, it can be seen that the electronic device provided in the embodiment of the present application receives the file selected by the user, performs parallel file transfer operations on the data upload module and the hash algorithm module, receives the file data through the data upload module, determines the corresponding data upload result, receives the file data through the hash algorithm module, inputs the file data into the set hash algorithm selection model, obtains the corresponding hash algorithm, performs a hash duplication detection operation according to the hash algorithm, the file data and the preset hash value database, determines the corresponding hash duplication result, receives the data upload result and the hash duplication result in parallel through the data processing module, and determines the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0248] In another embodiment, the file upload optimization method based on parallel computing can be configured separately from the central processing unit 9100. For example, the file upload optimization method based on parallel computing can be configured as a chip connected to the central processing unit 9100, and the function of the file upload optimization method based on parallel computing can be realized through the control of the central processing unit.

[0249] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0250] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0251] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0252] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0253] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0254] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0255] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0256] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0257] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the file upload optimization method based on parallel computing in the above-mentioned embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the file upload optimization method based on parallel computing in the above-mentioned embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0258] Step S101: receiving a file selected by a user through the parallel processing module, and performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine corresponding file data;

[0259] Step S102: receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0260] Step S103: The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload and storage result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0261] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application receives the file selected by the user, performs parallel file transfer operations on the data upload module and the hash algorithm module, receives the file data through the data upload module, determines the corresponding data upload result, receives the file data through the hash algorithm module, inputs the file data into the set hash algorithm selection model, obtains the corresponding hash algorithm, performs a hash duplication detection operation according to the hash algorithm, the file data and the preset hash value database, determines the corresponding hash duplication result, receives the data upload result and the hash duplication result in parallel through the data processing module, and determines the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0262] The embodiments of the present application also provide a computer program product capable of implementing all steps of the file upload optimization method based on parallel computing in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the file upload optimization method based on parallel computing are implemented. For example, the computer program / instruction implements the following steps:

[0263] Step S101: receiving a file selected by a user through the parallel processing module, and performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine corresponding file data;

[0264] Step S102: receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism;

[0265] Step S103: The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload and storage result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

[0266] From the above description, it can be seen that the computer program product provided by the embodiment of the present application receives the file selected by the user, performs parallel file transfer operations on the data upload module and the hash algorithm module, receives the file data through the data upload module, determines the corresponding data upload result, receives the file data through the hash algorithm module, inputs the file data into the set hash algorithm selection model, obtains the corresponding hash algorithm, performs a hash duplication detection operation according to the hash algorithm, the file data and the preset hash value database, determines the corresponding hash duplication result, receives the data upload result and the hash duplication result in parallel through the data processing module, and determines the corresponding upload database result according to the preset data processing rules, the data upload result and the hash duplication result, thereby improving the efficiency and flexibility of file upload based on intelligent hash algorithm selection and parallel computing methods.

[0267] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0268] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0269] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0270] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0271] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A file upload optimization method based on parallel computing, characterized in that: Applied to a file management system, the file management system includes a parallel processing module, a data upload module, a hash algorithm module and a data processing module, and the method includes: The parallel processing module receives the file selected by the user, and performs parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and the preset parallel processing rules to determine the corresponding file data; The file data is received through the data upload module, a data upload operation is performed on the file data, and a corresponding data upload result is determined; the file data is received through the hash algorithm module, the file data is input into a set hash algorithm selection model, a corresponding hash algorithm is obtained, a hash duplication detection operation is performed according to the hash algorithm, the file data and a preset hash value database, and a corresponding hash duplication result is determined, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism; The data upload result and the hash duplication result are received in parallel by the data processing module, and the corresponding upload to the warehouse result is determined according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload to the warehouse result includes file update storage or file termination storage.

2. The file upload optimization method based on parallel computing according to claim 1 is characterized in that: The performing parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and the preset parallel processing rule to determine the corresponding file data includes: Perform a parallel process start operation according to the file and preset parallel processing rules, and determine a corresponding parallel allocation path; According to the parallel allocation path, parallel file transfer operations are performed on the data upload module and the hash algorithm module respectively to determine corresponding file data.

3. The file upload optimization method based on parallel computing according to claim 2 is characterized in that: The performing a parallel process startup operation according to the file and the preset parallel processing rule to determine the corresponding parallel allocation path includes: Performing a data channel construction operation according to the file and a preset data access rule to determine a corresponding data access channel, wherein the preset data access rule includes at least one of shared memory, file mapping, and data stream; A parallel process startup operation is performed according to the data access channel and the preset parallel task scheduling rule to determine the corresponding parallel allocation path.

4. The file upload optimization method based on parallel computing according to claim 1 is characterized in that: Before receiving the file data through the hash algorithm module and inputting the file data into a set hash algorithm selection model to obtain a corresponding hash algorithm, the method includes: Acquire a historical file data set in the file management system, perform a feature extraction operation on the historical file data set according to a preset feature extraction rule, and determine a corresponding feature data set; Perform a labeling operation on the historical file data set according to a preset optimal hash tag to determine a corresponding label data set; A training set construction operation is performed on the feature data set and the label data set, and the training set obtained through the training set construction operation is input into a preset gradient boosting tree model for model training operation, and a corresponding set hash algorithm selection model is determined.

5. The file upload optimization method based on parallel computing according to claim 1 is characterized in that: The performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database to determine a corresponding hash duplication result includes: Performing a hash value calculation operation on the file data according to the hash algorithm to determine a corresponding file hash value; A hash duplication detection operation is performed according to the file hash value and a preset hash value database to determine a corresponding hash duplication result.

6. The file upload optimization method based on parallel computing according to claim 1 is characterized in that: The step of determining a corresponding upload and storage result according to a preset data processing rule, the data upload result, and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage, comprises: Determine whether the hash repetition result is a repetition; If so, the data upload module is subjected to a data termination upload instruction sending operation according to the preset synchronization mechanism, the user selection is obtained, and the corresponding upload and storage result is determined according to the user selection; If not, the file data is received to perform a database update operation on the database to determine the corresponding upload result.

7. The file upload optimization method based on parallel computing according to claim 1 is characterized in that: The method further comprises: determining a corresponding upload result according to a preset data processing rule, the data upload result and the hash duplication result, wherein the upload result includes file update storage or file termination storage, and further comprising: Determine whether the data upload result is upload completion; If completed, receiving the hash repetition result, and determining whether the hash repetition result is a repetition; If yes, obtain the user's selection, and determine the corresponding upload result according to the user's selection; If not, the file data is received to perform a database update operation on the database to determine the corresponding upload result.

8. A file upload optimization device based on parallel computing, characterized in that: The device comprises: A parallel processing module is used to receive a file selected by a user, and perform parallel file transfer operations on the data upload module and the hash algorithm module respectively according to the file and a preset parallel processing rule to determine the corresponding file data; A parallel computing module, comprising a data upload module and a hash algorithm module, for receiving the file data through the data upload module, performing a data upload operation on the file data, and determining a corresponding data upload result; receiving the file data through the hash algorithm module, inputting the file data into a set hash algorithm selection model, obtaining a corresponding hash algorithm, performing a hash duplication detection operation according to the hash algorithm, the file data, and a preset hash value database, and determining a corresponding hash duplication result, wherein the data upload module and the hash algorithm module are parallel process modules, connected by a preset synchronization mechanism; The data processing module is used to receive the data upload result and the hash duplication result in parallel, and determine the corresponding upload and storage result according to the preset data processing rules, the data upload result and the hash duplication result, wherein the upload and storage result includes file update storage or file termination storage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the file upload optimization method based on parallel computing described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the file upload optimization method based on parallel computing described in any one of claims 1 to 7 are implemented.

Images