Large file unloading method and system and storage medium

Abstract

The invention provides a large file unloading method, which is applied to a version management server and comprises the following steps of: receiving a version change file submitted by a user, and automatically detecting whether the version change file contains a large file conforming to a preset configuration rule or not; automatically transferring the identified large file to a third-party storage server, and deleting corresponding large file content in a version change file submitted by a user; and generating and storing a record file containing the transfer information. The invention further provides a corresponding system and a storage medium. By implementing the method and the device, the storage pressure of the version management tool during large file processing can be reduced, the downloading efficiency is improved, the user operation is simplified, and the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of version management, and in particular to a large file transfer method, system and storage medium applied to a version management server. Background Art

[0002] With the rapid development of the intelligent driving industry, algorithm software development capabilities have become one of the core competitive advantages of major car manufacturers. In the collaborative development of algorithm software in the field of intelligent driving, version management is a crucial practice. It can track and control different versions and changes of the project, ensure that team members can work together effectively, maintain code consistency, and quickly roll back or restore to any previous state, effectively ensuring code traceability, merge conflict resolution, and convenience of version release and distribution.

[0003] The existing mainstream version management tools include SVN, Git, etc., which have the following main problems in actual use: the tools themselves do not distinguish the file types uploaded by users, and all use incremental storage to manage version history. This means that if the user uploads compiled executable files, compressed files, video images and other non-text files, it is impossible to calculate the increment between each version change, and the version can only be managed in full. In the long run, the space occupied by the corresponding file space or code repository will increase exponentially, which will put pressure on the data server storage. At the same time, when the user pulls the corresponding project on the server from the client, the pulling time will be longer due to the large version history.

[0004] In the existing solutions, although installing various tool plug-ins (such as Git-lfs) can alleviate the problem to a certain extent, there are still some shortcomings:

[0005] Each user needs to install plug-ins locally on the client, edit configuration files, and have additional execution instructions for uploading and downloading. The configuration is complex and the learning cost is high.

[0006] In a multi-person collaborative team, if a member makes an incorrect submission due to improper configuration, the recent repository history needs to be rebuilt, which is costly to repair.

[0007] Users or administrators are required to actively pay attention to the large file upload records of the code repository. The tool side will not have any prompts for related large file uploads. Summary of the invention

[0008] The technical problem to be solved by the present invention is that the present invention proposes a large file transfer method, system and storage medium, which can reduce the storage pressure of version management tools when processing large files, improve download efficiency, simplify user operations and reduce maintenance costs.

[0009] As one aspect of the present invention, a large file transfer method is provided, which is applied in a version management server and at least comprises the following steps:

[0010] Receive version change files submitted by users and automatically detect whether they contain large files that meet the preset configuration rules;

[0011] Automatically transfer the identified large files to a third-party storage server and delete the corresponding large file content in the version change file submitted by the user;

[0012] Generate and store a record file containing the dumped information.

[0013] Among them, further include:

[0014] In the version management server, configuration rules for identifying large files are pre-configured using a visual configuration interface, and connection information of a third-party storage server is determined; the configuration rules include file type and / or file size threshold.

[0015] Among them, the automatic detection of whether the submitted content contains large files that meet the preset configuration rules includes:

[0016] Scan the version change files submitted by users and check the size and type of each file;

[0017] When it is detected that the version change file has a file type in the configuration rule, the file type is determined as a first-class large file;

[0018] When it is detected that the version change file exists in a file other than the first type of large files and exceeds the size threshold in the configuration rule, the user is asked whether to configure it as a large file for transfer; after receiving the user's confirmation, the file exceeding the size threshold is determined as a second type of large file;

[0019] The first category of large files and the second category of large files are determined as large files that comply with preset configuration rules.

[0020] The transfer information includes a download address link of the storage location of the transferred large file, and the generating and storing of a record file containing the transfer information further includes:

[0021] A text file with the same name as the version change file is generated and stored in the version management server. The text file with the same name also records the verification information and version information of the large file.

[0022] Among them, further include:

[0023] When it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold, the user is reminded to contact the administrator to add relevant configuration;

[0024] After the transfer is completed, the record file containing the transfer information is sent to the user.

[0025] Accordingly, as another aspect of the present invention, a large file transfer system is also included, which is applied to a version management server and at least includes:

[0026] The verification module is used to receive the version change files submitted by the user and automatically detect whether they contain large files that meet the preset configuration rules;

[0027] The transfer module is used to automatically transfer the identified large files to a third-party storage server and delete the corresponding large file content in the version change file submitted by the user;

[0028] The recording module is used to generate and store recording files containing transfer information.

[0029] Among them, further include:

[0030] The configuration module is used to pre-configure the configuration rules for identifying large files using a visual configuration interface, and to determine the connection information of a third-party storage server; the configuration rules include file type and / or file size threshold.

[0031] Wherein, the verification module further includes:

[0032] The scanning module is used to scan the version change files submitted by users and check the size and type of each file;

[0033] A first determination module, configured to determine a file of the file type as a first-category large file when it is detected that a version change file exists in a file type in the configuration rule;

[0034] The second determination module is used to, when checking that the version change file exists outside the first category of large files and exceeds the size threshold in the configuration rule, ask the user whether to configure it as a large file for transfer; after receiving the user's confirmation, determine the file exceeding the size threshold as a second category of large files;

[0035] The large file determination module is used to determine the first-category large files and the second-category large files as large files that comply with preset configuration rules.

[0036] Wherein, the recording module is further used for:

[0037] A text file with the same name as the version change file is generated and stored in the version management server, wherein the verification information and version information are also recorded in the text file with the same name; and the download address link of the storage location of the transferred large file is included in the transfer information.

[0038] Among them, further include:

[0039] The prompt processing unit is used to remind the user to contact the administrator to add relevant configuration when it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold; and after the transfer is completed, the record file containing the transfer information is sent to the user.

[0040] Correspondingly, as another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0041] Accordingly, as another aspect of the present invention, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computer device to perform operations corresponding to the aforementioned method.

[0042] Implementing the embodiments of the present invention has the following beneficial effects:

[0043] The present invention provides a large file transfer method, system and storage medium. The original complex user client configuration is centralized to the server segment for unified configuration by the administrator, and a visual interface is provided, which makes the operation very convenient. In the embodiment of the present invention, there is no need to install additional plug-ins or perform complex configuration locally, which saves the learning cost and configuration time of project members;

[0044] In the embodiment of the present invention, some large files in the version management file uploaded by the user are transferred to a third-party storage server, and the transfer and storage information records of the large files are stored in a text file; the storage pressure of the version management server is effectively reduced, and the download efficiency of the code repository containing the associated large files can be improved by storing the download address in the information record, reducing the user waiting time; at the same time, the administrator's later maintenance cost can also be reduced;

[0045] In the embodiment of the present invention, by providing a real-time reminder and feedback mechanism between the version management server and the user, problems caused by mistaken submission of large files can be avoided;

[0046] In summary, the implementation of the present invention can reduce the storage pressure of existing version management tools when processing large files, improve download efficiency, simplify user operations and reduce maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0048] Figure 1 A schematic diagram of the main process of an embodiment of a large file transfer method provided by the present invention;

[0049] Figure 2 Figure 1 A more detailed flowchart in ;

[0050] Figure 3 for Figure 2 Schematic diagram of the visual configuration interface in the version management server involved;

[0051] Figure 4 for Figure 2 Schematic diagram of the alarm pop-up window involved;

[0052] Figure 5 A schematic diagram of the structure of an embodiment of a large file transfer system provided by the present invention;

[0053] Figure 6 for Figure 5 Schematic diagram of the structure of the verification module. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 FIG. 1 is a schematic diagram showing the main flow of an embodiment of a large file transfer method provided by the present invention. Figures 2 to 4 As shown, in this embodiment, the method is applied in a version management server, which at least includes the following steps:

[0056] Step S10, pre-configuring a configuration rule for identifying large files in the version management server using a visual configuration interface, and determining connection information of a third-party storage server; the configuration rule includes a file type and / or a file size threshold, for example, a combination of at least one file type and a file size threshold;

[0057] Specifically, in the version management server, it is necessary to pre-configure which types of files need to be transferred as large files and set limits on the file sizes.

[0058] like Figure 3 As shown in the figure, a schematic diagram of a visual configuration interface is shown, in which you can select the "Yes" or "No" button to determine whether to enable the large file transfer function. At the same time, you can configure the file types that need to be transferred, such as ".zip", ".jpg", ".a" and other types of files in the figure, and the file named "filename.tar.gz"; at the same time, set the file size limit (i.e. size threshold) that needs to be alerted to 20MB.

[0059] Step S11, receiving the version change file submitted by the user, and automatically detecting whether it contains a large file that meets the preset configuration rules;

[0060] In a specific example, the step S11 further includes:

[0061] Scan the version change files submitted by users and check the size and type of each file;

[0062] When it is detected that the version change file has a file type in the configuration rule, the file of the file type is determined as a first-class large file; in a specific example, if the version change file is detected to have a file type in the configuration rule, the file type is determined as a first-class large file. Figure 3 If there are files of the types ".zip", ".jpg", ".a" and so on pre-configured in the system, or files named "filename.tar.gz", these files are regarded as the first type of large files;

[0063] When it is detected that the version change file is a file other than the first type of large file and exceeds the size threshold in the configuration rule, the user is asked whether to configure it as a large file for transfer; after receiving the user's confirmation, the file exceeding the size threshold is determined as a second type of large file; Figure 4 As shown, when it is detected that there are several files larger than 20MB (size threshold) in the version change file, an alarm pop-up window will be generated, which contains the specific information of the corresponding file: file name, path, size, etc., and is pushed to the user end for the user end to select. The user end can check some of them and request new configuration; or directly ignore the alarm information. If the user chooses to add new configuration, the version management server will update the file names selected by the user to the configuration rules, and at the same time identify these files as the second type of large files;

[0064] The first-category large files and the second-category large files are determined as large files that comply with preset configuration rules.

[0065] Step S12, automatically transferring the identified large files to a third-party storage server, and deleting the corresponding large file content in the version change file submitted by the user;

[0066] Specifically, in this step, the first type of large files with matching types or names and the second type of large files with excessive sizes confirmed by the user are removed from the current version management request; and then the two types of large files are uploaded to the third-party file storage server.

[0067] It is understandable that in other examples, the corresponding transfer operation may be started after the existence of the first type of large files is detected; after the transfer is completed, it is detected whether the second type of large files exists, and then the corresponding transfer operation is performed.

[0068] Step S13, generate and store a record file containing the transfer information. Then the version management server will put the adjusted version change management file into the version management warehouse so that other users can download and share it later.

[0069] In a specific example, the step S13 further includes:

[0070] A text file (such as a .txt file) with the same name as the version change file is generated and stored in the version management server, the transfer information includes the download address link of the transferred large file storage location, and the text file with the same name also records the verification information and version information of the large file. In a specific example, a record file is stored in the text file with the same name, and the record file at least includes: the download address link of the transferred large file storage location, verification information and version information. The verification information can be, for example, an MD5 checksum, which can check the correctness of the data by performing a hash operation on the received transmission data.

[0071] It can be understood that in the embodiment of the present invention, when it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold, the user is reminded to contact the administrator to add relevant configuration;

[0072] After the transfer is completed, the record file containing the transfer information is sent to the user.

[0073] It is understandable that during the version management process, if the version change file is pushed to other clients in the same project team, the updated version change file and the record file can be pushed together; other clients can quickly download the corresponding transferred large files from the third-party storage server according to the download address link in the record file.

[0074] It can be understood that in the embodiments of the present invention, secondary development is performed based on existing version management tools (such as Git): from the server-side configuration, designated non-text files (by file type, file name or file size) are distinguished to skip version management, and these files are transferred to a third-party storage server, while the version server only stores a text file that records the location of the third-party server where the file is stored and the file verification information, which can improve the download efficiency of code repositories containing associated large files.

[0075] In the present invention, individual users do not need to configure their own clients separately. Instead, the administrator can manage and configure the system in a unified manner, which saves the learning cost of project members and the space of the version management server. For a single code repository, users do not need to spend more time downloading or updating because of the large version history.

[0076] At the same time, the version server will verify each push request initiated by the user client. If there is a file that exceeds a certain size limit and does not exist in the server's configuration list, the user can be reminded to contact the administrator to add a new configuration type to avoid the situation where the file cannot be identified in time after being submitted by mistake, resulting in the need for the administrator to reshape the history and repair it.

[0077] In order to better understand the solution of the present invention, we can use a specific example to illustrate its workflow and innovations.

[0078] Suppose there is an intelligent driving algorithm development team that uses Git as a version management tool for collaborative development. During the development process, team members often need to upload some large files, such as model weight files (.pth), data sets (.zip or .tar.gz), and test videos (.mp4). If these large files are stored directly in the Git repository, the size of the repository will increase rapidly, affecting the download speed and storage efficiency of the repository.

[0079] Implementation plan content:

[0080] (1) The administrator configures the server

[0081] The administrator configures large file transfer rules on the Git server. The specific settings are as follows:

[0082] Enable large file transfer function: Enable this function for a specific repository.

[0083] Specify file types: Define file types such as .pth, .zip, .tar.gz, and .mp4 as large files.

[0084] File size quota: Set the file size threshold. For example, files larger than 20MB will be automatically considered large files.

[0085] (2) Automatic verification and transfer by the server

[0086] When team members submit version changes, the Git server automatically verifies whether these changes contain large files. For example, if a member submits a .zip file of 50MB in size, the server will recognize that this is a large file and automatically perform the following operations:

[0087] Exclude the .zip file from your Git commits.

[0088] Upload the file to a third-party file storage system (such as Amazon S3, Alibaba Cloud OSS, etc.).

[0089] (3) Record storage location and MD5 value

[0090] The Git server creates a .txt file with the same name as the .zip file in the version repository, which contains the following content:

[0091] Download address link: URL of the .zip file in the third-party storage system.

[0092] Version information: such as submission ID, submitter, etc.

[0093] MD5 checksum: MD5 checksum of the .zip file, used to verify file integrity.

[0094] During this process, project team members do not need to perform any additional configuration or operations when submitting version changes. They only need to submit code as usual, and the Git server will automatically handle the transfer and recording of large files in the background.

[0095] In the embodiment of the present invention, complex user client configurations are centralized on the server side, configured uniformly by the administrator, and a visual interface is provided. In this way, project members do not need to install additional plug-ins or perform complex configurations on each client, thus reducing learning costs.

[0096] The transfer, storage information recording and submission of large files are all automatically performed in the background, and users do not need to be aware of this process. This not only improves the user experience, but also reduces the cost of missubmission and repair caused by improper configuration.

[0097] By transferring large files to a third-party storage system and storing only the download address and MD5 checksum of the file in the Git repository, the storage pressure on the Git server is effectively reduced. At the same time, since the Git repository no longer contains large files, the download speed will also be significantly improved.

[0098] The server will perform real-time verification on each push request submitted by the user. If there is a file that exceeds the set size limit and is not in the configuration, the user will be reminded to contact the administrator to add new configuration. This avoids the situation where the warehouse history needs to be rebuilt after a large file is submitted by mistake, reducing the cost of repair.

[0099] In summary, the solution of the present invention has the advantages of centralized configuration, automated execution and real-time verification, effectively solves the storage and downloading problems of version management tools when processing large files, and improves development efficiency and team collaboration efficiency.

[0100] like Figure 5 FIG. 1 is a schematic diagram showing a structure of an embodiment of a large file transfer system provided by the present invention. Figure 6 As shown, in this embodiment, the system 1 is applied to a version management server, which at least includes:

[0101] A configuration module 10 is used to pre-configure a configuration rule for identifying large files using a visual configuration interface, and to determine connection information of a third-party storage server; the configuration rule includes a file type and / or a file size threshold, for example, a combination of at least one file type and a file size threshold;

[0102] The verification module 11 is used to receive the version change file submitted by the user and automatically detect whether it contains a large file that meets the preset configuration rules;

[0103] The transfer module 12 is used to automatically transfer the identified large files to a third-party storage server and delete the corresponding large file content in the version change file submitted by the user;

[0104] The recording module 13 is used to generate and store a recording file containing the transfer information.

[0105] like Figure 6 As shown, in a specific example, the verification module 11 further includes:

[0106] The scanning module 110 is used to scan the version change files submitted by the user and check the size and type of each file therein;

[0107] A first determination module 111 is used for determining a file of the file type as a first type of large file when checking that a version change file exists in a file type in the configuration rule;

[0108] The second determination module 112 is used to, when checking that the version change file exists outside the first category of large files and exceeds the size threshold in the configuration rule, ask the user whether to configure it as a large file for transfer; after receiving the user's confirmation, determine the file exceeding the size threshold as a second category of large files;

[0109] The large file determination module 113 is used to determine the first-category large files and the second-category large files as large files that comply with preset configuration rules.

[0110] In a more specific example, the recording module 13 is further used for:

[0111] A text file with the same name as the version change file is generated and stored in the version management server, and the verification information and version information are also recorded in the text file with the same name; the download address link of the storage location of the transferred large file is included in the transfer information. In a specific example, the record file is stored in the text file with the same name, and the record file at least includes: the download address link of the storage location of the transferred large file, the verification information and the version information.

[0112] It can be understood that, in a specific example, the system provided by the present invention further includes:

[0113] The prompt processing unit 14 is used to remind the user to contact the administrator to add relevant configuration when it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold; and after the transfer is completed, the record file containing the transfer information is sent to the user.

[0114] For more details, please refer to the above Figures 1 to 4 Description.

[0115] Accordingly, as another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figures 5 and 6 For more details, please refer to the above Figures 5 and 6 Description.

[0116] Accordingly, as another aspect of the present invention, a computer program product is also provided, comprising computer instructions, wherein the computer instructions instruct a computer device to execute the above-mentioned Figures 5 and 6 For more details, please refer to the above Figures 5 and 6 Description.

[0117] Implementing the embodiments of the present invention has the following beneficial effects:

[0118] The present invention provides a large file transfer method, system and storage medium. The original complex user client configuration is centralized to the server segment for unified configuration by the administrator, and a visual interface is provided, which makes the operation very convenient. In the embodiment of the present invention, there is no need to install additional plug-ins or perform complex configuration locally, which saves the learning cost and configuration time of project members;

[0119] In the embodiment of the present invention, some large files in the version management file uploaded by the user are transferred to a third-party storage server, and the transfer and storage information records of the large files are stored in a text file; the storage pressure of the version management server is effectively reduced, and the download efficiency of the code repository containing the associated large files can be improved by storing the download address in the information record, and the user waiting time can be reduced; at the same time, the administrator's later maintenance cost can also be reduced, and there is no need to perform operations such as repairing the warehouse expansion problem caused by large files and reshaping the history as in the prior art;

[0120] In the embodiment of the present invention, by providing a real-time reminder and feedback mechanism between the version management server and the user, problems caused by mistaken submission of large files can be avoided;

[0121] In summary, the implementation of the present invention can reduce the storage pressure of existing version management tools when processing large files, improve download efficiency, simplify user operations and reduce maintenance costs.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A large file transfer method, applied in a version management server, characterized in that: The following steps are involved: Receive version change files submitted by users and automatically detect whether they contain large files that meet the preset configuration rules; Automatically transfer the identified large files to a third-party storage server and delete the corresponding large file content in the version change file submitted by the user; Generate and store a record file containing the dumped information.

2. The method according to claim 1, characterized in that Further including: In the version management server, use the visual configuration interface to pre-configure the configuration rules for identifying large files and determine the connection information of the third-party storage server; The configuration rule includes a file type and / or a file size threshold.

3. The method according to claim 1, characterized in that Automatically detect whether the submitted content contains large files that meet the pre-set configuration rules, including: Scan the version change files submitted by users and check the size and type of each file; When it is detected that the version change file has a file type in the configuration rule, the file type is determined as a first-class large file; When it is detected that the version change file exists in a file other than the first type of large files and exceeds the size threshold in the configuration rule, the user is asked whether to configure it as a large file for transfer; after receiving the user's confirmation, the file exceeding the size threshold is determined as a second type of large file; The first-category large files and the second-category large files are determined as large files that comply with preset configuration rules.

4. The method according to claim 1, characterized in that The transfer information includes a download address link of the storage location of the transferred large file, and the generating and storing of a record file including the transfer information further includes: A text file with the same name as the version change file is generated and stored in the version management server. The text file with the same name also records the verification information and version information of the large file.

5. The method according to claim 1, characterized in that Further including: When it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold, the user is reminded to contact the administrator to add relevant configuration; After the transfer is completed, the record file containing the transfer information is sent to the user.

6. A large file transfer system, applied to a version management server, characterized in that: include: The verification module is used to receive the version change files submitted by the user and automatically detect whether they contain large files that meet the preset configuration rules; The transfer module is used to automatically transfer the identified large files to a third-party storage server and delete the corresponding large file content in the version change file submitted by the user; The recording module is used to generate and store recording files containing transfer information.

7. The system according to claim 6, characterized in that Further including: The configuration module is used to pre-configure the configuration rules for identifying large files using a visual configuration interface, and to determine the connection information of a third-party storage server; the configuration rules include file type and / or file size threshold.

8. The system according to claim 6, characterized in that The verification module further comprises: The scanning module is used to scan the version change files submitted by users and check the size and type of each file; A first determination module, configured to determine a file of the file type as a first-category large file when it is detected that a version change file exists in a file type in the configuration rule; The second determination module is used to, when checking that the version change file exists outside the first category of large files and exceeds the size threshold in the configuration rule, ask the user whether to configure it as a large file for transfer; after receiving the user's confirmation, determine the file exceeding the size threshold as a second category of large files; The large file determination module is used to determine the first-category large files and the second-category large files as large files that comply with preset configuration rules.

9. The system according to claim 6, characterized in that The recording module is further used for: A text file with the same name as the version change file is generated and stored in the version management server, wherein the text file with the same name also records the verification information and version information; and the transfer information includes a download address link of the storage location of the transferred large file.

10. The system according to claim 6, characterized in that Further including: The prompt processing unit is used to remind the user to contact the administrator to add relevant configuration when it is detected that the version change file provided by the user is not included in the server configuration but the size exceeds the predetermined size threshold; and after the transfer is completed, the record file containing the transfer information is sent to the user.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

12. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computer device to perform operations corresponding to the method according to any one of claims 1 to 5.

Images