Large file copying method and device, electronic equipment and storage medium

Abstract

The invention provides a large file copying method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: responding to a kernel command under the condition that the file size of a to-be-processed file is greater than a preset threshold value, and responding to the kernel command under the condition that a source file address and a destination file address of the to-be-processed file meet a preset address condition; obtaining a queue state of a preset annular queue; and under the condition that the queue state meets the first state condition, sequentially extracting file contents of the to-be-processed file from the source file address and sequentially storing the file contents into an annular queue, and under the condition that the queue state meets the second state condition, sequentially extracting the file contents stored in the annular queue and sequentially writing the file contents into a target file address. The method and the device are all realized in a kernel layer, and interaction between the kernel layer and an application layer is not involved, so that the system overhead is reduced. In addition, for copying of the large file, the process of caching data of the large file to the memory is omitted, and the stability of the file caching system is improved.

Description

[0001] This application is a divisional application of the parent case with application number "202411124925.7", application date "2024.08.16", and invention name "Large file copying method, device, electronic device and storage medium" Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a large file copying method, device, electronic device and computer-readable storage medium. Background Art

[0003] File copying is an important function of computer systems. How to efficiently complete file copying is an important factor in improving computer performance.

[0004] Currently, when a file system copies a file, it can first copy the file's data content to the application layer through the kernel's read interface, and cache the file's data content to the memory. Then, it calls the kernel's write interface to write the application layer's data content to the destination address, repeating this read and write process to finally complete the file copy.

[0005] However, in the current solution, when the size of the file to be copied is large, the execution process needs to copy data back and forth between the kernel and the application layer, resulting in excessive system overhead. In addition, due to the limited capacity of the memory, the caching action of large files during the copying process can easily affect the stability of the file cache. Summary of the invention

[0006] The embodiments of the present application provide a large file copying method and device to solve the problems in the prior art.

[0007] In a first aspect, an embodiment of the present application provides a large file copying method, the method comprising: When the file size of the file to be processed is greater than a preset threshold, a preset kernel command is called; In response to the kernel command, and when it is determined that the source file address and the destination file address of the to-be-processed file meet a preset address condition, a queue state of a preset ring queue is obtained; the queue state is used to characterize a situation in which the ring queue stores data; When the queue state satisfies a first state condition, the file contents of the to-be-processed files are sequentially extracted from the source file address and sequentially stored in the circular queue; and when the queue state satisfies a second state condition, the file contents stored in the circular queue are sequentially extracted and sequentially written into the destination file address; When the queue status satisfies the termination condition, the copying of the file to be processed is completed.

[0008] In a second aspect, an embodiment of the present application provides a large file copying device, the device comprising: A calling module, used for calling a preset kernel command when the file size of the file to be processed is greater than a preset threshold; A state module, for responding to the kernel command and obtaining a queue state of a preset circular queue when it is determined that the source file address and the destination file address of the file to be processed meet a preset address condition; the queue state is used to characterize the situation of storing data in the circular queue; a copy module, configured to extract the file contents of the to-be-processed files from the source file address in sequence and store them in the circular queue in sequence when the queue state satisfies a first state condition, and to extract the file contents stored in the circular queue in sequence and write them in sequence to the destination file address in sequence when the queue state satisfies a second state condition; The termination module is used to complete the copying of the file to be processed when the queue status meets the termination condition.

[0009] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method of the first aspect.

[0010] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method of the first aspect.

[0011] In the case where the file size of the to-be-processed file is greater than a preset threshold, the embodiment of the present application executes a preset kernel command, and in the case where it is determined that the source file address and the destination file address of the to-be-processed file meet the preset address condition, obtains the queue state of the preset circular queue, and in the case where the queue state meets the first state condition, sequentially extracts the file content of the to-be-processed file from the source file address and sequentially stores it in the circular queue, and in the case where the queue state meets the second state condition, sequentially extracts the file content stored in the circular queue and sequentially writes it to the destination file address, and performs the read-write process back and forth to complete the copying of the to-be-processed file. The entire large file copying process of the embodiment of the present application is implemented at the kernel layer, and does not involve the interaction between the kernel layer and the application layer, thereby reducing the system overhead. In addition, the copying of large files also saves the process of caching the data of large files to the memory, reduces the pressure of the file cache system, and improves the stability of the file cache system.

[0012] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 This is a flowchart of a large file copying method provided by an embodiment of the present application; Figure 2 This is a flowchart of the specific steps of a large file copying method provided by an embodiment of the present application; Figure 3 is a schematic diagram of a ring queue provided in an embodiment of the present application; Figure 4 It is a parallel processing flow chart provided by an embodiment of the present application; Figure 5 This is a flowchart of a large file copy process provided by an embodiment of the present application; Figure 6 It is a serial processing flow chart provided by an embodiment of the present application; Figure 7 This is a block diagram of a large file copying device provided in an embodiment of the present application; Figure 8 is a block diagram of an apparatus of the present application; Fig. 9 It is a schematic diagram of the structure of the server in some embodiments of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims is used to describe the association relationship of associated objects, indicating that three kinds of relationships can exist, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the front and back associated objects are a kind of "or" relationship. In the embodiment of the present application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0017] Figure 1 , is a flowchart of a large file copying method provided by an embodiment of the present application, such as Figure 1 As shown, the method may include: Step 101: When the file size of the file to be processed is greater than a preset threshold, a preset kernel command is called.

[0018] Since the related art needs to copy data back and forth between the kernel and the application layer during the process of copying files, and involves data cache operations, this results in large system overhead and memory instability when copying large files. The embodiment of the present application aims to optimize the copy process of large files and solve the above problems generated when copying large files.

[0019] First, the embodiment of the present application can analyze the pending file waiting to be copied to determine whether the pending file is a large file. Specifically, the embodiment of the present application can set a preset threshold and obtain the file size of the pending file. When the file size of the pending file is greater than the preset threshold, it is determined that the pending file is a large file. Among them, the preset threshold can be set according to actual needs. For example, the preset threshold can be set to 6GB. Then, when the file size of the pending file is greater than 6GB, the pending file can be determined to be a large file.

[0020] Furthermore, the embodiment of the present application develops a new kernel command for copying large files: copy_file_direct. This command can directly copy large files from the kernel layer, eliminating the data interaction between the application layer and the kernel layer. In addition, in the process of executing this command to copy large files, it is possible to bypass the file system's cache system, that is, the data content of the large file is not cached in the memory.

[0021] Specifically, calling the kernel command may be to implement an application binary interface (ABI, Application Binary Interface) call, where ABI is a standard that defines an interface between an application program and an operating system or hardware.

[0022] Step 102: In response to the kernel command, and when it is determined that the source file address and the destination file address of the file to be processed meet the preset address condition, obtain the queue state of the preset ring queue; the queue state is used to characterize the situation of storing data in the ring queue.

[0023] In the embodiment of the present application, the source file address of the to-be-processed file represents the location of the to-be-processed file before copying, and the destination file address of the to-be-processed file represents the location of the to-be-processed file after copying. The preset address condition represents that the source file address and the destination file address of the to-be-processed file meet certain special conditions. When the conditions are met, the subsequent copying process can be accelerated through parallel read and write operations without generating negative effects.

[0024] Specifically, the embodiment of the present application can set a circular queue in the kernel to replace the application layer to maintain the data used in the parallel execution of reading and writing through the circular queue. In response to the kernel command, the embodiment of the present application can first obtain the queue state of the circular queue when it is determined that the source file address and the destination file address of the file to be processed meet the preset address condition. The queue state is used to characterize the situation of storing data in the circular queue. Then, the read / write process can control itself to be in a dormant or working state according to the queue state until the file copying is completed in parallel.

[0025] Optionally, the preset address condition is that the source file address and the destination file address are not in the same disk.

[0026] In one implementation, the preset address condition may indicate that the source file address and the destination file address are on the same disk. This is because the head of a mechanical disk has a seek overhead, that is, it takes a certain amount of time for the head to move to a specific position. If the read-write parallel copy strategy is adopted when the source file address and the destination file address are on the same disk, the head of the disk needs to move back and forth repeatedly, which in turn causes a decrease in performance.

[0027] When the source file address and the destination file address are not on the same disk, a read-write parallel copy strategy is adopted. For example, when the source file address and the destination file address are on two different disks, the heads of the two disks can work in parallel. Since the heads of the two disks move independently, there is no mutual interference between the two, thus meeting the requirements for parallel execution of read and write operations and effectively improving the copying efficiency of large files.

[0028] Step 103: When the queue state satisfies the first state condition, the file contents of the files to be processed are extracted from the source file address in sequence and stored in the circular queue in sequence; and when the queue state satisfies the second state condition, the file contents stored in the circular queue are extracted in sequence and written into the destination file address in sequence.

[0029] In the embodiment of the present application, the circular queue can be composed of multiple memory pages, each memory page represents a memory area, and the memory page is used to store data content. The function of the read process is to extract the file content of the file to be processed from the source file address in sequence and store it in the circular queue in sequence. As long as there are empty memory pages (memory pages without data) in the circular queue, the read process can work; the function of the write process is to extract the file content stored in the circular queue and write it to the destination file address in sequence. As long as there are non-empty memory pages (memory pages with data) in the circular queue, the write process can work. The two work in parallel and cooperate to complete the copying of the file.

[0030] Specifically, if the queue state satisfies the first state condition, it can be indicated that the state of the circular queue can satisfy the work of the reading process, that is, there are empty memory pages in the circular queue; if the queue state satisfies the second state condition, it can be indicated that the state of the circular queue can satisfy the work of the writing process, that is, there are non-empty memory pages in the circular queue. The reading process and the writing process can work in parallel independently according to their corresponding state conditions to copy the data.

[0031] For example, the reading process can start from the starting position of a large file, such as reading the data content of several pages, and start from the starting page of the circular queue and store the read data content in sequence; the writing process can start from the starting page of the circular queue, extract the data content stored in the memory pages of the circular queue in sequence, and write them into the destination file address in sequence. The above process can be repeated to complete the copying of the file.

[0032] Step 104, when the queue status satisfies the termination condition, the copying of the to-be-processed file is completed.

[0033] In the embodiment of the present application, the termination condition indicates that the data content of the file to be processed is completely copied from the source file address to the destination file address, thereby generating a new file obtained after the copy at the destination file address, and completing the copy of the file to be processed. The entire large file copying process of the embodiment of the present application is implemented at the kernel layer, and does not involve the interaction between the kernel layer and the application layer. In addition, the copying of large files also omits the process of caching the data of large files into the memory, reducing the pressure on the file cache system and improving the stability of the file cache system.

[0034] In summary, the embodiment of the present application can execute a preset kernel command when the file size of the to-be-processed file is greater than a preset threshold, and obtain the queue state of the preset circular queue when it is determined that the source file address and the destination file address of the to-be-processed file meet the preset address condition; when the queue state meets the first state condition, the file content of the to-be-processed file is sequentially extracted from the source file address and stored in the circular queue in sequence, and when the queue state meets the second state condition, the file content stored in the circular queue is sequentially extracted and written to the destination file address in sequence, and the read-write process is performed back and forth to complete the copying of the to-be-processed file. The entire large file copying process of the embodiment of the present application is implemented at the kernel layer, and does not involve the interaction between the kernel layer and the application layer, thereby reducing the system overhead. In addition, the copying of large files also omits the process of caching the data of large files to the memory, reducing the pressure of the file cache system and improving the stability of the file cache system.

[0035] Figure 2 , is a flowchart of the specific steps of a large file copying method provided in an embodiment of the present application, such as Figure 2 As shown, the method may include: Step 201: When the file size of the file to be processed is greater than a preset threshold, a preset kernel command is called.

[0036] This step may be specifically referred to as step 101 and will not be described in detail here.

[0037] Optionally, the preset threshold is the product of the total memory capacity and a preset coefficient, and the preset coefficient may be 0.7.

[0038] In the embodiment of the present application, in general, the memory of the device is a fixed element, and the total memory capacity is a fixed value. Since caching large files will affect the working stability of the memory. Therefore, the product of the total memory capacity and the preset coefficient can be used as a preset threshold, so that the to-be-processed files whose file size is greater than the preset threshold can be identified as large files. Preferably, the preset coefficient can be 0.7. Of course, the value of the preset coefficient can also be adjusted according to actual needs, such as 0.6, 0.8, etc.

[0039] Step 202: In response to the kernel command, and when it is determined that the source file address and the destination file address of the file to be processed meet the preset address condition, obtain the queue state of the preset ring queue; the queue state is used to characterize the situation of storing data in the ring queue.

[0040] This step may be specifically referred to as step 102 and will not be described in detail here.

[0041] Step 203: When the queue state indicates that there are empty memory pages in the circular queue, the file contents of the to-be-processed files are extracted from the source file addresses in sequence through the read process, and are stored in the empty memory pages in sequence; the empty memory pages represent memory pages that do not store data contents.

[0042] The circular queue includes a plurality of memory pages; the reading process is used to sequentially extract the file contents of the to-be-processed files from the source file addresses and sequentially store them in the circular queue.

[0043] Step 204: When the queue state indicates that there is no empty memory page in the circular queue, control the reading process to sleep.

[0044] In the embodiment of the present application, for steps 203-204, the circular queue may be composed of multiple memory pages, each memory page represents a memory slice, and the memory page is used to store data content. Figure 3 As shown, it shows a schematic diagram of a circular queue. Assuming that the circular queue includes 8 memory pages, the read process is used to write the read data in the order of memory pages 1-8; the write process is used to extract data in the order of memory pages 1-8.

[0045] The queue state represents the data storage status of the memory pages in the circular queue. The queue state can reflect whether the circular queue includes empty memory pages (memory pages without data), and whether the circular queue includes non-empty memory pages (memory pages with data).

[0046] In one case, a memory page may be 4k in size and the ring queue may be 128 megabytes in size.

[0047] For the reading process, as long as the queue status indicates that there are empty memory pages in the circular queue (that is, the capacity of the circular queue is still available), the reading process can work normally and write the read data content to the empty memory page. When the queue status indicates that there are no empty memory pages in the circular queue, the reading process cannot work normally and the reading process can enter sleep waiting.

[0048] Specifically, the reading process can extract the file contents of the to-be-processed files from the source file addresses in sequence through the block device interface provided by the kernel. The block device interface is an interface provided for storage devices such as hard disks and optical drives. It is a bottom-level interface used to directly read and write file data without going through the application layer.

[0049] For example, refer to Figure 3, the first flag (HEAD) reflects the memory page to which the reading process in the circular queue currently writes data, that is, the reading process starts writing the read data content from memory page 1 in sequence until the data content of memory page 5 is written. At this time, the queue state can indicate that there are still memory pages 6, 7, 8 and other memory pages that do not store data, so the read request can continue to work and write data to memory pages 6, 7, and 8 in sequence until the queue state indicates that there are no empty memory pages at present, and the read request enters the dormant state.

[0050] Further references Figure 4 , which shows a read-write parallel processing flow chart of an embodiment of the present application: S1. Create a reading process.

[0051] S2. The data content of the file to be processed is sequentially extracted from the source file address using the block device interface through the reading process, and the read data content is sequentially stored in the memory page starting from the starting position of the circular queue.

[0052] S3. The write thread starts from the starting position of the circular queue, reads the data content stored in the memory page in sequence, and uses the block device interface to write the data content to the destination file address in sequence.

[0053] Among them, step S2 and step S3 are executed in parallel, thereby improving the efficiency of file copying. According to tests, in the process of copying large files from a disk to a USB flash drive, parallel execution of reading and writing can save 30% of the time.

[0054] Step 205: When the queue state indicates that there are non-empty memory pages in the circular queue, the file contents stored in the non-empty memory pages are sequentially extracted by the write process and written into the destination file addresses sequentially, and the non-empty memory pages represent memory pages storing data contents.

[0055] The circular queue includes a plurality of memory pages; the writing process is used to sequentially extract the file contents stored in the circular queue and sequentially write them into the destination file addresses.

[0056] Step 206: When the queue state indicates that the non-empty memory page does not exist in the circular queue, control the writing process to sleep.

[0057] For the writing process, as long as the queue status indicates that there are non-empty memory pages in the circular queue (that is, there is data content in the circular queue), the writing process can work normally and perform the operation of extracting data content from the non-empty memory page. When the queue status indicates that there are no non-empty memory pages in the circular queue, the writing process cannot work normally and the writing process can enter sleep waiting.

[0058] Specifically, the writing process can write the file content to the destination file address through the block device interface provided by the kernel. The block device interface is an interface provided for storage devices such as hard disks and optical drives. It is a low-level interface used to directly read and write file data without going through the application layer.

[0059] For example, refer to Figure 3 , the second flag (TAIL) reflects the memory page from which the write process in the circular queue currently extracts data content, that is, the write process extracts data content from memory page 1 in sequence until the data content of memory page 2 is extracted. At this time, the queue state can indicate that there are still memory pages 3, 4, 5 and other memory pages with data, then the write request can continue to work, extracting data from memory pages 3, 4, and 5 in sequence, until the queue state indicates that there are no non-empty memory pages at present, and the write request enters the dormant state.

[0060] Step 207: When it is detected that the read process generates an end flag, the write process writes all the file contents stored in the remaining non-empty memory pages in the circular queue to the destination file address, and then completes the copying of the file to be processed. The non-empty memory page represents a memory page with data content.

[0061] Among them, the reading process is used to extract the file content of the file to be processed from the source file address in sequence and store them in the circular queue in sequence, and the writing process is used to extract the file content stored in the circular queue in sequence and write them into the destination file address in sequence; the end flag is generated by the reading process when the last data page in the source file address is read.

[0062] In the embodiment of the present application, the file to be processed includes multiple data pages, and the reading process reads the data content of the file to be processed in sequence from the first data page until the last data page is read, and the reading process generates an end flag. The writing process can identify the end flag, and the writing process can write the file content stored in all the remaining non-empty memory pages in the circular queue to the destination file address according to the end flag, and then determine that the copy of the file to be processed is completed, that is, all the data pages of the file to be processed are copied from the source file address to the destination file address.

[0063] Optionally, the method may further include: Step 208: When it is determined that the source file address and the destination file address of the to-be-processed file do not satisfy the preset address condition, apply for a memory buffer of a preset size.

[0064] Step 209: extract the file contents of the to-be-processed files from the source file addresses in sequence, and store them in the memory buffer in sequence.

[0065] Step 210: When the memory buffer is full, extract the file contents stored in the memory buffer in sequence and write them into the destination file address in sequence.

[0066] Step 211, when the file content is not stored in the memory buffer, enter the step of extracting the file content of the to-be-processed file from the source file address in sequence and storing it in the memory buffer in sequence until all the file content of the source file address is extracted.

[0067] In the embodiment of the present application, for steps 208 to 211, the source file address of the to-be-processed file represents the location of the to-be-processed file before copying, and the destination file address of the to-be-processed file represents the location of the to-be-processed file after copying. The preset address condition represents that the source file address and the destination file address of the to-be-processed file meet certain special conditions. When the condition is met, the subsequent copying process can be accelerated through parallel read and write operations without generating negative effects.

[0068] Specifically, the preset address condition is that the source file address and the destination file address are not in the same disk. When the source file address and the destination file address of the file to be processed meet the preset address condition, the read and write operations can be performed in parallel, thereby improving the efficiency of copying large files. However, since the head of the mechanical disk has a seek overhead, that is, it takes a certain amount of movement time for the head to move to a specific position, when the source file address and the destination file address of the file to be processed do not meet the preset address condition, that is, the source file address and the destination file address are in the same disk, if the read-write parallel copy strategy is adopted, it will cause the head of the disk to move back and forth repeatedly, which will cause a decrease in performance.

[0069] In order to solve this problem, the embodiment of the present application can adopt a read-write serial copy strategy when the source file address and the destination file address of the to-be-processed file do not meet the preset address conditions. Figure 5 As shown: M1. Make sure the file to be processed is a large file.

[0070] M2. Call kernel commands.

[0071] M3. Determine whether the source file address and the destination file address meet the preset address conditions.

[0072] If yes, execute M4 and execute the parallel strategy.

[0073] If not, execute M5 to execute the serial strategy.

[0074] Specifically, the serial copy strategy is to first apply for a memory buffer of a preset size for use, and the memory buffer can be in the form of a circular queue or a non-circular queue. Afterwards, the processing process sequentially extracts the file content of the to-be-processed file from the source file address and stores it in the memory buffer in sequence. When the memory buffer is full, the processing process sequentially extracts the file content stored in the memory buffer and writes it to the destination file address in sequence. Afterwards, when all the file content in the memory buffer is taken out, the processing process sequentially extracts the file content of the to-be-processed file from the source file address and stores it in the memory buffer in sequence. This cycle is repeated until all the file content of the source file address is copied to the destination file address.

[0075] The entire serial execution process can be referred to Figure 6 ,include: N1. Apply for a memory buffer of preset size.

[0076] N2. When the memory buffer is empty, the processing process extracts the file contents of the files to be processed from the source file addresses in sequence and stores them in the memory buffer in sequence.

[0077] N3. When the memory buffer is full, the processing process sequentially extracts the file contents stored in the memory buffer and writes them into the destination file address sequentially.

[0078] N4. Determine whether all the file contents of the source file address have been read. If so, exit; otherwise, enter N2.

[0079] In summary, in the embodiment of the present application, when the file size of the to-be-processed file is greater than a preset threshold, the preset kernel command is executed, and when it is determined that the source file address and the destination file address of the to-be-processed file meet the preset address condition, the queue state of the preset circular queue is obtained, and when the queue state meets the first state condition, the file content of the to-be-processed file is sequentially extracted from the source file address and stored in the circular queue in sequence, and when the queue state meets the second state condition, the file content stored in the circular queue is sequentially extracted and written to the destination file address in sequence, and the read-write process is performed back and forth to complete the copying of the to-be-processed file. The entire large file copying process of the embodiment of the present application is implemented at the kernel layer, and does not involve the interaction between the kernel layer and the application layer, thereby reducing the system overhead. In addition, the copying of large files also saves the process of caching the data of large files to the memory, reduces the pressure of the file cache system, and improves the stability of the file cache system.

[0080] Figure 7 : is a block diagram of a large file copying device provided in an embodiment of the present application, the device comprising: The calling module 301 is used to call a preset kernel command when the file size of the file to be processed is greater than a preset threshold; The state module 302 is used to respond to the kernel command and obtain the queue state of the preset ring queue when it is determined that the source file address and the destination file address of the file to be processed meet the preset address condition; the queue state is used to characterize the situation of storing data in the ring queue; The copy module 303 is used to extract the file contents of the to-be-processed files from the source file address in sequence and store them in the circular queue in sequence when the queue state satisfies the first state condition, and to extract the file contents stored in the circular queue in sequence and write them in sequence to the destination file address in sequence when the queue state satisfies the second state condition; The termination module 304 is used to complete the copying of the to-be-processed file when the queue status satisfies the termination condition.

[0081] Optionally, the device further comprises: An application module, used for applying for a memory buffer of a preset size when it is determined that the source file address and the destination file address of the to-be-processed file do not meet the preset address condition; A first processing module, used for sequentially extracting the file contents of the to-be-processed files from the source file addresses and sequentially storing them in the memory buffer; A second processing module, configured to extract the file contents stored in the memory buffer in sequence and write them into the destination file address in sequence when the memory buffer is full; The third processing module is used to enter the step of extracting the file content of the to-be-processed file from the source file address in sequence and storing it in the memory buffer in sequence when the file content is not stored in the memory buffer, until all the file content of the source file address is extracted.

[0082] Optionally, the preset address condition is that the source file address and the destination file address are not located in the same disk.

[0083] Optionally, the circular queue includes a plurality of memory pages; the reading process is used to sequentially extract the file contents of the to-be-processed files from the source file addresses and sequentially store them in the circular queue; The copy module 303 includes: A reading submodule, configured to extract the file contents of the to-be-processed files from the source file addresses in sequence through the reading process and store them in sequence into the empty memory pages when the queue state indicates that there are empty memory pages in the circular queue; the empty memory pages indicate memory pages that do not store data contents; The first sleep submodule is used to control the reading process to sleep when the queue state indicates that there is no empty memory page in the circular queue.

[0084] Optionally, the circular queue includes multiple memory pages; the writing process is used to sequentially extract the file contents stored in the circular queue and write them into the destination file addresses sequentially.

[0085] The copy module 303 includes: A writing submodule, used for extracting the file contents stored in the non-empty memory pages and writing them into the destination file addresses in sequence through the writing process when the queue state indicates that there are non-empty memory pages in the circular queue, wherein the non-empty memory pages indicate memory pages storing data contents; The second sleep submodule is used to control the writing process to sleep when the queue state indicates that the non-empty memory page does not exist in the circular queue.

[0086] Optionally, the circular queue includes a plurality of memory pages; the termination module 304 includes: The termination submodule is used for completing the copying of the to-be-processed file after the to-be-written process writes all the file contents stored in the remaining non-empty memory pages in the circular queue to the destination file address when it is detected that the reading process has generated an end flag; the non-empty memory pages represent memory pages storing data contents; Among them, the reading process is used to extract the file content of the file to be processed from the source file address in sequence and store them in the circular queue in sequence, and the writing process is used to extract the file content stored in the circular queue in sequence and write them into the destination file address in sequence; the end flag is generated by the reading process when the last data page in the source file address is read.

[0087] Optionally, the preset threshold is the product of the total memory capacity and a preset coefficient, and the preset coefficient is 0.7.

[0088] In summary, in the embodiment of the present application, when the file size of the to-be-processed file is greater than a preset threshold, the preset kernel command is executed, and when it is determined that the source file address and the destination file address of the to-be-processed file meet the preset address condition, the queue state of the preset circular queue is obtained, and when the queue state meets the first state condition, the file content of the to-be-processed file is sequentially extracted from the source file address and stored in the circular queue in sequence, and when the queue state meets the second state condition, the file content stored in the circular queue is sequentially extracted and written to the destination file address in sequence, and the read-write process is performed back and forth to complete the copying of the to-be-processed file. The entire large file copying process of the embodiment of the present application is implemented at the kernel layer, and does not involve the interaction between the kernel layer and the application layer, thereby reducing the system overhead. In addition, the copying of large files also saves the process of caching the data of large files to the memory, reduces the pressure of the file cache system, and improves the stability of the file cache system.

[0089] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0090] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0091] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0092] An embodiment of the present application provides a large file copying device, including a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors to perform the methods described in one or more of the above embodiments.

[0093] Figure 8 8 is a block diagram of a large file copying device 800 according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0094] Reference Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0095] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0096] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0097] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0098] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0099] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0100] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0101] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also search for changes in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0102] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0103] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0104] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0105] Fig. 9 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (e.g., one or more processors) and memory 1932, and one or more storage media 1930 (e.g., one or more mass storage devices) storing application programs 1942 or data 1944. The memory 1932 and storage medium 1930 may be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules ( Fig. 9 Each module may include a series of instruction operations on the server. Furthermore, the central processor 1922 may be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0106] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (a server or a terminal), enables the device to execute the method of the above embodiment.

[0108] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the device (server or terminal), the device can perform the description of the method of the above embodiment, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0109] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions so that the computer device performs the description of the method of the above embodiment, so it will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0110] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0111] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0112] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0113] The above is a detailed introduction to a large file copying method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A large file copying method, characterized in that: The method comprises: When the file size of the file to be processed is greater than a preset threshold, a preset kernel command is called; In response to the kernel command, and when it is determined that the source file address and the destination file address of the to-be-processed file meet the preset address condition, the queue state of the preset circular queue is obtained; the queue state is used to characterize the situation of the circular queue storing data; the circular queue includes a plurality of memory pages; the write process is used to sequentially extract the file contents stored in the circular queue and sequentially write them into the destination file address through the block device interface provided by the kernel; the read process is used to sequentially extract the file contents of the to-be-processed file from the source file address through the block device interface provided by the kernel and sequentially store them into the circular queue; In the case where the queue state indicates that there are empty memory pages in the circular queue, the file contents of the to-be-processed files are sequentially extracted from the source file address by the reading process, and are sequentially stored in the empty memory pages; the empty memory pages represent memory pages that do not store data contents; in the case where the queue state indicates that there are no empty memory pages in the circular queue, the reading process is controlled to be dormant, and in the case where the queue state indicates that there are non-empty memory pages in the circular queue, the file contents stored in the non-empty memory pages are sequentially extracted by the writing process and are sequentially written to the destination file address, the non-empty memory pages represent memory pages that store data contents; in the case where the queue state indicates that there are no non-empty memory pages in the circular queue, the writing process is controlled to be dormant; When the queue status satisfies the termination condition, the copying of the file to be processed is completed.

2. The large file copying method according to claim 1, characterized in that: The method further comprises: When it is determined that the source file address and the destination file address of the to-be-processed file do not satisfy the preset address condition, applying for a memory buffer of a preset size; Extracting the file contents of the to-be-processed files from the source file addresses in sequence, and storing them in the memory buffer in sequence; When the memory buffer is full, extracting the file contents stored in the memory buffer in sequence and writing them into the destination file address in sequence; In the case that the file content is not stored in the memory buffer, the step of extracting the file content of the to-be-processed file from the source file address in sequence and storing them in the memory buffer in sequence is entered until all the file content of the source file address is extracted.

3. The large file copying method according to claim 1 or 2, characterized in that: The preset address condition is that the source file address and the destination file address are not located in the same disk.

4. The large file copying method according to claim 1, characterized in that: The ring queue includes a plurality of memory pages; When the queue state satisfies the termination condition, completing the copying of the file to be processed includes: When it is detected that the read process generates an end flag, the write process completes the copying of the to-be-processed file after writing all the file contents stored in the remaining non-empty memory pages in the circular queue to the destination file address; the non-empty memory pages represent memory pages storing data contents; Among them, the reading process is used to extract the file content of the file to be processed from the source file address in sequence and store them in the circular queue in sequence, and the writing process is used to extract the file content stored in the circular queue in sequence and write them into the destination file address in sequence; the end flag is generated by the reading process when the last data page in the source file address is read.

5. The large file copying method according to claim 1, characterized in that: The preset threshold is the product of the total memory capacity and a preset coefficient, and the preset coefficient is 0.

7.

6. A large file copying device, characterized in that: The device comprises: A calling module, used for calling a preset kernel command when the file size of the file to be processed is greater than a preset threshold; A state module is used to respond to the kernel command and obtain a queue state of a preset circular queue when it is determined that the source file address and the destination file address of the file to be processed meet the preset address condition; the queue state is used to characterize the situation of storing data in the circular queue; the circular queue includes multiple memory pages; the write process is used to sequentially extract the file content stored in the circular queue and sequentially write it to the destination file address through the block device interface provided by the kernel; the read process is used to sequentially extract the file content of the file to be processed from the source file address through the block device interface provided by the kernel and sequentially store it in the circular queue; A copy module, used for, when the queue state indicates that there are empty memory pages in the circular queue, extracting the file contents of the to-be-processed file from the source file address in sequence through the read process, and storing them in sequence into the empty memory pages; the empty memory pages represent memory pages without data contents; when the queue state indicates that there are no empty memory pages in the circular queue, controlling the read process to sleep, and when the queue state indicates that there are non-empty memory pages in the circular queue, extracting the file contents stored in the non-empty memory pages in sequence through the write process and writing them in sequence into the destination file address, the non-empty memory pages represent memory pages with data contents; when the queue state indicates that there are no non-empty memory pages in the circular queue, controlling the write process to sleep; The termination module is used to complete the copying of the file to be processed when the queue status meets the termination condition.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 5.

Images