Windows large folder efficient copying method based on Golang coroutine

Abstract

The invention discloses a Windows large folder efficient copying method based on Golang coroutines, and belongs to the technical field of computer file copying and multi-terminal backup, and the Windows large folder efficient copying method based on the Golang coroutines comprises the steps of multi-task concurrence based on the Golang coroutines: utilizing a coroutine mechanism of the Golang to complete read-write operation of each file by an independent coroutine so as to realize real parallel processing; intelligent analysis and compression: analyzing whether the file carries redundant drawings or not, automatically compressing the large file exceeding a preset threshold value by the system, and segmenting the compressed file into a plurality of small blocks for transmission, in the invention, efficient concurrent processing is realized, concurrent processing of file reading and writing is realized through a Golang coroutine mechanism, and the file reading and writing efficiency is improved. According to the method, the copying efficiency is greatly improved, intelligent compression and fragmentation are realized, intelligent compression and fragmentation processing is carried out on large files, the transmitted data volume is reduced, the transmission speed is increased, multi-terminal equipment supports and supports multiple pieces of equipment to participate in file copying, and network resources are fully utilized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer file copying and multi-terminal backup, and in particular relates to a Windows large folder efficient copying method based on Golang coroutine. Background Art

[0002] Operating system API calls: Most file copy tools implement file copy functions by calling the file read and write APIs provided by the operating system. Single-threaded processing means that file reading and writing are usually completed in one thread. This mode limits the speed of processing large files.

[0003] The existing Windows large folder efficient copy method based on Golang coroutine has the following technical defects:

[0004] Inefficiency: Single-threaded processing cannot fully utilize the capabilities of modern multi-core processors;

[0005] Resource waste: During the file transfer process, CPU and other hardware resources are not used efficiently;

[0006] Network latency: When it comes to copying across a network, the transfer speed is severely limited due to single-threaded limitations;

[0007] Difficulty in handling large files: For very large files or folders, traditional copying methods are prone to performance bottlenecks.

[0008] Based on this, the present invention designs a Windows large folder efficient copy method based on Golang coroutine to solve the above problems. Summary of the invention

[0009] The purpose of the present invention is to solve the problems of low efficiency, resource waste, network delay and difficulty in processing large files in the existing Windows large folder efficient copy method based on Golang coroutine, and to propose a Windows large folder efficient copy method based on Golang coroutine.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A Windows large folder efficient copy method based on Golang coroutine, including:

[0012] Multi-task concurrency based on Golang coroutine: Using Golang's coroutine mechanism, the reading and writing operations of each file are completed by a separate coroutine to achieve true parallel processing;

[0013] Intelligent analysis and compression: Analyze whether the file contains redundant drawings. For large files exceeding the preset threshold, the system will automatically compress them and split the compressed files into multiple small blocks for transmission;

[0014] High-performance multi-device transmission: supports multiple devices to participate in the file copy process at the same time, and uses network bandwidth for efficient file transmission;

[0015] Initialization phase: Analyze all files in the target folder and decide whether to compress and fragment them according to the file size and type;

[0016] Task allocation: Create corresponding coroutines to process each file based on the number and size of the files;

[0017] File processing: Each coroutine is responsible for reading a file and writing it to the target location. For files that need to be compressed and fragmented, they are first compressed and then fragmented.

[0018] Progress monitoring: monitor the progress of all tasks in real time and update the user interface display;

[0019] Error handling: If an error occurs during the copy process, the error information is logged and an attempt is made to recover or skip the file and continue copying other files.

[0020] As a further description of the above technical solution:

[0021] Analyze and check whether the documents contain any extra drawings, including:

[0022] Divide the copied files into common image type files, document files and model drawing files, and analyze them in different ways respectively;

[0023] For common image type files, the image is displayed directly;

[0024] For document files, since this type of file may contain a large number of drawing images, it is necessary to first retrieve all images contained in this type of file, and then display the retrieved images in sequence;

[0025] For model drawing files, it is necessary not only to display this type of drawing, but also to analyze the information inside the drawing.

[0026] As a further description of the above technical solution:

[0027] The internal information analysis of the model drawing file includes:

[0028] Preprocess the model drawing files, first convert the model drawing files into image format, then use Gaussian filtering technology to remove noise in the image, and combine threshold processing technology and maximum inter-class variance method to remove non-information parts;

[0029] Since the amount of information in the model drawings is usually large, it is not conducive to direct recognition, so the image needs to be segmented and the information is segmented using image slicing technology;

[0030] The content proposes that for the segmented information fragments, a table structure recognition model is used to extract the information of each small block.

[0031] As a further description of the above technical solution:

[0032] In the image segmentation process, the image is divided into multiple small blocks, and each small block is inferred separately, so as to solve the problem of loss of detail information after image scaling and improve the detection effect of information.

[0033] As a further description of the above technical solution:

[0034] The table structure recognition model is a Cycle-CenterNet model, which adopts a bottom-up approach:

[0035] Based on the center point of the cell, regress the distances of the four vertices and decode the cell;

[0036] Based on the cell vertices, regress the distance to the center point of the cells sharing the vertex and decode the combo box;

[0037] Based on the combo box, the discrete bounding boxes are spliced ​​to obtain the accurately completed electronic information;

[0038] The calculation formulas for the three indicators of precision, recall and score include:

[0039]

[0040] In the formula, XY is the correctly extracted data, CW is the incorrectly extracted data, and CN is the data that could not be extracted.

[0041] As a further description of the above technical solution:

[0042] The accuracy, recall rate and score of the Cycle-CenterNet model are evaluated. The accuracy rate is the ratio of correctly extracted data in the extracted information data, the recall rate is the ratio of correctly extracted data in all the information data to be extracted, and the score is used to comprehensively evaluate the recall rate and accuracy rate.

[0043] As a further description of the above technical solution:

[0044] In the intelligent analysis and compression, each file in a large folder is analyzed one by one, and useless information in the file is identified through specific algorithms and logics, and the useless information is identified and removed to optimize file storage and transmission efficiency.

[0045] As a further description of the above technical solution:

[0046] The file compression specifically includes:

[0047] Trigger condition: when the file size exceeds the preset threshold, the system will automatically trigger the compression mechanism. This threshold is usually set according to system configuration and user needs.

[0048] Compression method: Use efficient compression algorithm to compress files. The size of compressed files will be significantly reduced, which is used to save storage space and speed up network transmission.

[0049] Fragmentation processing, in order to further improve the transmission efficiency, the compressed file will be divided into multiple small pieces, these pieces can be transmitted in parallel, making full use of network bandwidth resources.

[0050] As a further description of the above technical solution:

[0051] The efficient compression algorithm uses gzip, bzip2 or a more efficient custom algorithm. The compression algorithm effectively reduces the file size by reducing redundant data in the file.

[0052] As a further description of the above technical solution:

[0053] The compression ratio is a key indicator to measure the effectiveness of the compression algorithm, and is usually calculated using the following formula:

[0054] Compression ratio = (original data size - compressed data size) / original data size × 100%

[0055] Or in some cases:

[0056] Compression ratio = original data size / compressed data size × 100%. If the compression ratio calculated in this way is greater than 100%, it means that the data is compressed, and the larger the value, the more obvious the compression effect. However, in actual use, the former is more intuitive and is used to reflect the ability of the compression algorithm to reduce data size.

[0057] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0058] 1. In the present invention, efficient concurrent processing is achieved through the Golang coroutine mechanism to realize concurrent processing of file reading and writing, which greatly improves the copying efficiency. Intelligent compression and fragmentation are performed on large files, which reduces the amount of data transmitted and speeds up the transmission speed. Multi-terminal device support supports multiple devices to participate in file copying and makes full use of network resources.

[0059] 2. In the present invention, the copying speed is significantly improved. Through multi-task concurrent processing, the file copying time is greatly shortened, the network bandwidth occupancy is reduced, large files are compressed and fragmented, the amount of data transmitted over the network is effectively reduced, and the user experience is improved. Real-time progress monitoring allows users to clearly understand the copying status, enhancing the convenience of use.

[0060] 3. In the present invention, an image retrieval and analysis method is used to more accurately retrieve and analyze the file content when copying a large folder, and to determine the illegal copied files. The application effect shows that the time required for staff to check files is effectively reduced.

[0061] 4. In the present invention, the model drawing files are preprocessed to optimize the drawing quality, provide a basis for subsequent data extraction and analysis, and ensure the efficiency and accuracy of subsequent analysis results.

[0062] 5. In the present invention, by converting the model drawing file into an image format and applying advanced image processing technology and table structure recognition model, the recognition accuracy and processing efficiency of the model drawing file data can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of an efficient copy method for large Windows folders based on Golang coroutines proposed by the present invention;

[0064] Figure 2 This is a schematic diagram of the architecture of a Windows large folder efficient copy method based on Golang coroutine proposed by the present invention;

[0065] Figure 3 This is a schematic diagram of the image retrieval process in a Windows large folder efficient copy method based on Golang coroutine proposed by the present invention;

[0066] Figure 4 A schematic diagram of a process of analyzing and checking whether a file contains extra drawings in a Windows large folder efficient copy method based on Golang coroutine proposed by the present invention;

[0067] Figure 5A schematic diagram of the internal information analysis process of a model drawing file in a Windows large folder efficient copy method based on Golang coroutine proposed by the present invention;

[0068] Figure 6 The present invention provides a flowchart of file compression in a Windows large folder efficient copy method based on Golang coroutine. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Please see attached Figure 1 -Attached Figure 6 The present invention provides a technical solution: a Windows large folder efficient copy method based on Golang coroutine, comprising:

[0071] Multi-task concurrency based on Golang coroutine: Using Golang's coroutine mechanism, the reading and writing operations of each file are completed by a separate coroutine to achieve true parallel processing;

[0072] Intelligent analysis and compression: Analyze whether the file contains redundant drawings. For large files exceeding the preset threshold, the system will automatically compress them and split the compressed files into multiple small blocks for transmission;

[0073] High-performance multi-device transmission: supports multiple devices to participate in the file copy process at the same time, and uses network bandwidth for efficient file transmission;

[0074] Initialization phase: Analyze all files in the target folder and decide whether to compress and fragment them according to the file size and type;

[0075] Task allocation: Create corresponding coroutines to process each file based on the number and size of the files;

[0076] File processing: Each coroutine is responsible for reading a file and writing it to the target location. For files that need to be compressed and fragmented, they are first compressed and then fragmented.

[0077] Progress monitoring: monitor the progress of all tasks in real time and update the user interface display;

[0078] Error handling: If an error occurs during the copy process, the error information is logged and an attempt is made to recover or skip the file and continue copying other files.

[0079] As a further description of the above technical solution:

[0080] Analyze and check whether the documents contain any extra drawings, including:

[0081] Divide the copied files into common image type files, document files and model drawing files, and analyze them in different ways respectively;

[0082] For common image type files, the image is displayed directly;

[0083] For document files, since this type of file may contain a large number of drawing images, it is necessary to first retrieve all images contained in this type of file, and then display the retrieved images in sequence;

[0084] For model drawing files, it is necessary not only to display this type of drawing, but also to analyze the information inside the drawing.

[0085] As a further description of the above technical solution:

[0086] The internal information analysis of the model drawing file includes:

[0087] Preprocess the model drawing files, first convert the model drawing files into image format, then use Gaussian filtering technology to remove noise in the image, and combine threshold processing technology and maximum inter-class variance method to remove non-information parts;

[0088] Since the amount of information in the model drawings is usually large, it is not conducive to direct recognition, so the image needs to be segmented and the information is segmented using image slicing technology;

[0089] The content proposes that for the segmented information fragments, a table structure recognition model is used to extract the information of each small block.

[0090] As a further description of the above technical solution:

[0091] In the image segmentation process, the image is divided into multiple small blocks, and each small block is inferred separately, so as to solve the problem of loss of detail information after image scaling and improve the detection effect of information.

[0092] As a further description of the above technical solution:

[0093] The table structure recognition model is a Cycle-CenterNet model, which adopts a bottom-up approach:

[0094] Based on the center point of the cell, regress the distances of the four vertices and decode the cell;

[0095] Based on the cell vertices, regress the distance to the center point of the cells sharing the vertex and decode the combo box;

[0096] Based on the combo box, the discrete bounding boxes are spliced ​​to obtain the accurately completed electronic information;

[0097] The calculation formulas for the three indicators of precision, recall and score include:

[0098]

[0099] In the formula, XY is the correctly extracted data, CW is the incorrectly extracted data, and CN is the data that could not be extracted.

[0100] As a further description of the above technical solution:

[0101] The accuracy, recall rate and score of the Cycle-CenterNet model are evaluated. The accuracy rate is the ratio of correctly extracted data in the extracted information data, the recall rate is the ratio of correctly extracted data in all the information data to be extracted, and the score is used to comprehensively evaluate the recall rate and accuracy rate.

[0102] As a further description of the above technical solution:

[0103] In the intelligent analysis and compression, each file in a large folder is analyzed one by one, and useless information in the file is identified through specific algorithms and logics, and the useless information is identified and removed to optimize file storage and transmission efficiency.

[0104] As a further description of the above technical solution:

[0105] The file compression specifically includes:

[0106] Trigger condition: when the file size exceeds the preset threshold, the system will automatically trigger the compression mechanism. This threshold is usually set according to system configuration and user needs.

[0107] Compression method: Use efficient compression algorithm to compress files. The size of compressed files will be significantly reduced, which is used to save storage space and speed up network transmission.

[0108] Fragmentation processing, in order to further improve the transmission efficiency, the compressed file will be divided into multiple small pieces, these pieces can be transmitted in parallel, making full use of network bandwidth resources.

[0109] As a further description of the above technical solution:

[0110] The efficient compression algorithm uses gzip, bzip2 or a more efficient custom algorithm. The compression algorithm effectively reduces the file size by reducing redundant data in the file.

[0111] As a further description of the above technical solution:

[0112] The compression ratio is a key indicator to measure the effectiveness of the compression algorithm, and is usually calculated using the following formula:

[0113] Compression ratio = (original data size - compressed data size) / original data size × 100%

[0114] Or in some cases:

[0115] Compression ratio = original data size / compressed data size × 100%. If the compression ratio calculated in this way is greater than 100%, it means that the data is compressed, and the larger the value, the more obvious the compression effect. However, in actual use, the former is more intuitive and is used to reflect the ability of the compression algorithm to reduce data size.

[0116] Working principle, when using:

[0117] Multi-task concurrency based on Golang coroutine: Using Golang's coroutine mechanism, the reading and writing operations of each file are completed by a separate coroutine to achieve true parallel processing;

[0118] Intelligent analysis and compression: Analyze whether the file contains redundant drawings. For large files exceeding the preset threshold, the system will automatically compress them and split the compressed files into multiple small blocks for transmission;

[0119] Analyze and check whether the documents contain any extra drawings, including:

[0120] Divide the copied files into common image type files, document files and model drawing files, and analyze them in different ways respectively;

[0121] For common image type files, the image is displayed directly;

[0122] For document files, since this type of file may contain a large number of drawing images, it is necessary to first retrieve all images contained in this type of file, and then display the retrieved images in sequence;

[0123] For model drawing files, it is necessary not only to display the drawings of this type, but also to analyze the information inside the drawings;

[0124] The internal information analysis of the model drawing file includes:

[0125] Preprocess the model drawing files, first convert the model drawing files into image format, then use Gaussian filtering technology to remove noise in the image, and combine threshold processing technology and maximum inter-class variance method to remove non-information parts;

[0126] Since the amount of information in the model drawings is usually large, it is not conducive to direct recognition. Therefore, the image needs to be segmented and processed. The image slicing technology is used to segment the information. In the image segmentation process, the image is divided into multiple small blocks, and each small block is inferred separately to solve the problem of loss of detail information after image scaling, so as to improve the detection effect of information;

[0127] The content proposes that for the information fragments after segmentation, a table structure recognition model is used to extract the information of each small block. The table structure recognition model is a Cycle-CenterNet model. The Cycle-CenterNet model adopts a bottom-up approach:

[0128] Based on the center point of the cell, regress the distances of the four vertices and decode the cell;

[0129] Based on the cell vertices, regress the distance to the center point of the cells sharing the vertex and decode the combo box;

[0130] Based on the combo box, the discrete bounding boxes are spliced ​​to obtain the accurately completed electronic information;

[0131] The accuracy, recall rate and score of the Cycle-CenterNet model are evaluated. The accuracy rate is the ratio of correctly extracted data in the extracted information data, the recall rate is the ratio of correctly extracted data in all the information data to be extracted, and the score is used to comprehensively evaluate the recall rate and accuracy rate;

[0132] In the intelligent analysis and compression, each file in a large folder is analyzed one by one, and useless information in the file is identified and removed through specific algorithms and logics to optimize file storage and transmission efficiency;

[0133] The file compression specifically includes:

[0134] Trigger condition: when the file size exceeds the preset threshold, the system will automatically trigger the compression mechanism. This threshold is usually set according to system configuration and user needs.

[0135] Compression method: Use efficient compression algorithm to compress files. The size of compressed files will be significantly reduced, which is used to save storage space and speed up network transmission.

[0136] Fragmentation processing: To further improve transmission efficiency, the compressed file will be divided into multiple small pieces. These pieces can be transmitted in parallel to make full use of network bandwidth resources.

[0137] High-performance multi-device transmission: supports multiple devices to participate in the file copy process at the same time, and uses network bandwidth for efficient file transmission;

[0138] Initialization phase: Analyze all files in the target folder and decide whether to compress and fragment them according to the file size and type;

[0139] Task allocation: Create corresponding coroutines to process each file based on the number and size of the files;

[0140] File processing: Each coroutine is responsible for reading a file and writing it to the target location. For files that need to be compressed and fragmented, they are first compressed and then fragmented.

[0141] Progress monitoring: monitor the progress of all tasks in real time and update the user interface display;

[0142] Error handling: If an error occurs during the copy process, the error information is logged and an attempt is made to recover or skip the file and continue copying other files.

[0143] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Windows large folder efficient copy method based on Golang coroutine, characterized in that: include: Multi-task concurrency based on Golang coroutine: Using Golang's coroutine mechanism, the reading and writing operations of each file are completed by a separate coroutine to achieve true parallel processing; Intelligent analysis and compression: Analyze whether the file contains redundant drawings. For large files exceeding the preset threshold, the system will automatically compress them and split the compressed files into multiple small blocks for transmission; High-performance multi-device transmission: supports multiple devices to participate in the file copy process at the same time, and uses network bandwidth for efficient file transmission; Initialization phase: Analyze all files in the target folder and decide whether to compress and fragment them according to the file size and type; Task allocation: Create corresponding coroutines to process each file based on the number and size of the files; File processing: Each coroutine is responsible for reading a file and writing it to the target location. For files that need to be compressed and fragmented, they are first compressed and then fragmented. Progress monitoring: monitor the progress of all tasks in real time and update the user interface display; Error handling: If an error occurs during the copy process, the error information is logged and an attempt is made to recover or skip the file and continue copying other files.

2. According to claim 1, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: Analyze and check whether the documents contain any extra drawings, including: Divide the copied files into common image type files, document files and model drawing files, and analyze them in different ways respectively; For common image type files, the image is displayed directly; For document files, since this type of file may contain a large number of drawing images, it is necessary to first retrieve all images contained in this type of file, and then display the retrieved images in sequence; For model drawing files, it is necessary not only to display this type of drawing, but also to analyze the information inside the drawing.

3. According to claim 2, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: The internal information analysis of the model drawing file includes: Preprocess the model drawing files, first convert the model drawing files into image format, then use Gaussian filtering technology to remove noise in the image, and combine threshold processing technology and maximum inter-class variance method to remove non-information parts; Since the amount of information in the model drawings is usually large, it is not conducive to direct recognition, so the image needs to be segmented and the information is segmented using image slicing technology; The content proposes that for the segmented information fragments, a table structure recognition model is used to extract the information of each small block.

4. According to claim 3, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: In the image segmentation process, the image is divided into multiple small blocks, and each small block is inferred separately, so as to solve the problem of loss of detail information after image scaling and improve the detection effect of information.

5. According to claim 4, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: The table structure recognition model is a Cycle-CenterNet model, which adopts a bottom-up approach: Based on the center point of the cell, regress the distances of the four vertices and decode the cell; Based on the cell vertices, regress the distance to the center point of the cells sharing the vertex and decode the combo box; Based on the combo box, the discrete bounding boxes are spliced ​​to obtain the accurately completed electronic information.

6. According to claim 5, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: The accuracy, recall rate and score of the Cycle-CenterNet model are evaluated. The accuracy rate is the ratio of correctly extracted data in the extracted information data, the recall rate is the ratio of correctly extracted data in all the information data to be extracted, and the score is used to comprehensively evaluate the recall rate and accuracy rate.

7. According to claim 6, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: In the intelligent analysis and compression, each file in a large folder is analyzed one by one, and useless information in the file is identified through specific algorithms and logics, and the useless information is identified and removed to optimize file storage and transmission efficiency.

8. According to claim 7, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: The file compression specifically includes: Trigger condition: when the file size exceeds the preset threshold, the system will automatically trigger the compression mechanism. This threshold is usually set according to system configuration and user needs. Compression method: Use efficient compression algorithm to compress files. The size of compressed files will be significantly reduced, which is used to save storage space and speed up network transmission. Fragmentation processing, in order to further improve the transmission efficiency, the compressed file will be divided into multiple small pieces, these pieces can be transmitted in parallel, making full use of network bandwidth resources.

9. According to claim 8, a Windows large folder efficient copy method based on Golang coroutine is characterized in that: The efficient compression algorithm uses gzip, bzip2 or a more efficient custom algorithm. The compression algorithm effectively reduces the file size by reducing redundant data in the file.

10. A Windows large folder efficient copy method based on Golang coroutine according to claim 9, characterized in that: The compression ratio is a key indicator to measure the effectiveness of the compression algorithm, and is usually calculated using the following formula: Compression ratio = (original data size - compressed data size) / original data size × 100% Or in some cases: Compression ratio = original data size / compressed data size × 100%. If the compression ratio calculated in this way is greater than 100%, it means that the data is compressed, and the larger the value, the more obvious the compression effect. However, in actual use, the former is more intuitive and is used to reflect the ability of the compression algorithm to reduce data size.

Images