A file fragment uploading method based on bastion machine operation and maintenance

Abstract

The application discloses a file fragment uploading method based on a bastion machine operation and maintenance, a bastion machine server accepts file fragments, and stores the file fragments into Caffeine cache, Redis cache and a database; after successful uploading, SHA1 value of the fragments is calculated, and MD5 of the fragments is calculated; if the MD5 value is consistent with MD5 value of fragments transmitted from the front end, the uploading state of the fragments in the Caffeine cache, the Redis cache and the database is changed to uploading success. The bastion machine server sequentially queries the Caffeine cache, the Redis cache and the database, finds all uploaded fragments belonging to the file, performs file fragment merging, calculates MD5 value of the file after successful merging, and changes the uploading state of the file in the database to uploading success if the MD5 value is consistent with MD5 value of uploading parameters. The application can view the file download during security audit, and enriches management and control of the bastion machine on a managed host.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of file transmission, and particularly relates to a file fragment uploading method based on a bastion machine operation and maintenance. BACKGROUND

[0002] The bastion machine is an operation and maintenance and security audit management and control platform, can centrally manage asset permissions, whole-process controls operation behaviors, restores operation and maintenance scenes in real time, guarantees that cloud operation and maintenance behaviors are identifiable, permissions are controllable, and operations are auditable, solves problems such as difficult management of numerous assets, unclear operation and maintenance responsibility permissions, and difficult tracing of operation and maintenance events, and helps enterprises to meet the regulatory requirements of the Cybersecurity Law. In the existing bastion machine security products in China, most of them realize operation and maintenance and audit functions of asset types such as SSH operation and maintenance, RDP operation and maintenance, and database operation and maintenance. When performing SSH and RDP host operation and maintenance, users can upload local files to the operation and maintenance host, and administrators can download the uploaded files for auditing in the audit, but if the uploaded file is large, there will be performance problems. In the existing operation and maintenance scene of the bastion machine, if a client uploads a large file to a managed host, it will occupy a large amount of bandwidth resources, causing the browser to be avoided from being stuck; if the browser is refreshed, the uploading will be interrupted, and needs to be re-uploaded, which is not good for user experience. SUMMARY

[0003] The application aims to provide a file fragment uploading method based on a bastion machine operation and maintenance, and aims to solve the above problems. The application can be used to realize storage of uploaded files when using a bastion machine H5 operation and maintenance, and can download and view the files during security auditing, enriches the management and control of the bastion machine on the managed host, and can comprehensively manage and protect the client assets. The application can solve the pain point of large file uploading occupying a large amount of bandwidth resources, causing the browser to be avoided from being stuck.

[0004] The application mainly realizes the following technical scheme:

[0005] A file fragment uploading method based on a bastion machine operation and maintenance, comprising the following steps:

[0006] Step S100: uploading a file and reading the file name and file size of the file, and calculating the SHA1 value of the file;

[0007] Step S200: dividing the total size of the file by a set size, and rounding up the result, to calculate the total number of file fragments;

[0008] Step S300: the bastion machine server queries a database by using the parameters of the file in step S100 and step S200, if the database does not match the same file, writes the file in the database, and sets the file uploading state field to uploading;

[0009] Step S400: the front end divides the file into slices according to a set size, records the serial number of each slice, calculates the SHA1 value of the slice, and calculates the MD5 of the slice by taking the SHA1 value of the slice, the file name and the serial number as salt values;

[0010] Step S500: the bastion host server sequentially queries the server Caffeine cache, Redis cache and database according to the parameters of the slice in step S400, and if none of the caches matches the slice, it is indicated that the slice has not been uploaded, the slice data is written into the Caffeine cache, and the upload state of the slice is set to uploading.

[0011] Step S600: the front end calls the upload interface, uploads the slice MD5, slice serial number, slice content and file name to the bastion host server, the bastion host server accepts the file slice, stores the slice into the Caffeine cache, Redis cache and database, calculates the SHA1 value of the slice after successful uploading, and calculates the MD5 of the slice by taking the SHA1 value of the slice, the file name and the serial number of the slice as salt values; if the MD5 value of the slice transmitted by the front end is consistent, it is indicated that the slice content is uploaded correctly, and the upload state of the slice in the Caffeine cache, Redis cache and database is changed to uploading success.

[0012] Step S700: the front end calls the file merging interface, uploads the file name and file SHA1 value to the bastion host server as parameters, the bastion host server sequentially queries the Caffeine cache, Redis cache and database, finds all the slices of the file that have been uploaded, performs file slice merging, calculates the MD5 value of the file after successful merging, and if the MD5 value is consistent with the uploading parameter, the file is uploaded successfully and a message is sent to the front end, all slice caches are deleted, and the upload state of the file in the database is changed to uploading success.

[0013] In order to better implement the application, further, in step S300, the file checking interface is called, the file name, file size, total number of file slices and file SHA1 value are uploaded to the bastion host server, and the bastion host server queries the database according to the four parameters.

[0014] In order to better implement the application, further, in step S300, if the database matches the same file, the upload state of the file is checked; if the upload state is uploading failure, it is indicated that the file upload record already exists but the uploading is not completed, step S400 is continuously executed for reuploading; if the upload state is uploaded, it is indicated that the file has been successfully uploaded to the server, and the server directly sends a file uploaded successfully message to the client.

[0015] For better implementation of the present application, further, the step S500 comprises: the front end sequentially calls the file fragment checking interface according to the sequence number of the fragments, uploads the MD5 of each fragment, the sequence number of the fragment and the file name to the bastion host server, and the bastion host server sequentially queries the server Caffeine cache, Redis cache and database according to the three parameters.

[0016] For better implementation of the present application, further, in the step S500, if the same fragment is matched in any one of the Caffeine cache, Redis cache and database, the uploading state of the fragment is checked; if the uploading state is uploading failure, it is indicated that the file uploading record already exists but the uploading is not completed, so the fragment needs to be re-uploaded, and the step S600 is executed; if the uploading state is uploading success, it is indicated that the fragment has been successfully uploaded to the server, if this is the last fragment, the step S700 is executed; otherwise, the step S500 is executed to check the next fragment.

[0017] For better implementation of the present application, further, in the step S600, if the server detects network disconnection or calculates that the MD5 value is inconsistent with the uploading parameter MD5 value, the fragment uploading fails, the uploading state of the fragment in the Caffeine cache, Redis cache and database is changed to uploading failure, and the uploading state of the file is changed to uploading failure.

[0018] For better implementation of the present application, further, in the step S100, the file name and file size of the file are read by using the Blob object of Javascript; and in the step S400, the file is fragmented by using the slice method of the Blob object.

[0019] The present application has the following beneficial effects:

[0020] (1) The present application firstly uses SHA1 algorithm to judge file uniqueness, the collision probability of SHA1 algorithm is less than that of MD5, and SHA1 is a safer and more reliable algorithm; secondly, the present application uses MD5 with salt, uses the SHA1 value of the fragment, the file name and the sequence number of the fragment as the salt value, and calculates the MD5 of each fragment, which has three advantages:

[0021] ① For large file fragments, the performance is higher by using MD5 calculation;

[0022] ② Using the three values as part of the salt value is equivalent to calculating the accuracy of SHA1 and MD5, and further improving the accuracy of the uniqueness of the file fragment;

[0023] III. Use SHA1 as salt value to calculate MD5, which can greatly improve the security of MD5, and the possibility of MD5 being parsed and collided is greatly reduced.

[0024] (2) When the file is uploaded, the front end does not know the uploading state of the file and each slice of the file, and whether the file and the uploading state of the file are queried from the database according to step S300, if the file has been uploaded, the file does not need to be uploaded again, if the file does not exist or the file fails to be uploaded, the uploading state of each slice is sequentially queried from the Caffeine cache, the Redis cache and the database value according to step S500, if the uploading state of the slice is uploaded, it indicates that the slice of the file has been successfully uploaded before, but other slices may not be successfully uploaded or the file slice merging fails to cause the whole file to fail to be uploaded, then the file uploading this time can skip the slice, so that the uploading time can be saved, and the effect of breakpoint resume is achieved.

[0025] (3) When the slice is saved, the two-level cache of the Caffeine cache + the Redis cache is used, the Caffeine cache is read and written first, the Caffeine cache is a local cache compared with the Redis cache, and the reading and writing speed is faster and the efficiency is higher, if the Caffeine cache is not matched, the Redis cache is queried, and finally the database is queried if needed, so that the reading and writing times of the database are greatly reduced, the reading and writing times of the Redis cache are also reduced, and the storage efficiency of the server is improved. DETAILED DESCRIPTION

[0026] Embodiment 1:

[0027] A file slice uploading method based on a bastion machine operation and maintenance, comprising the following steps:

[0028] Step S100: uploading a file and reading the file name and file size of the file by using a Blob object of Javascript, and calculating the SHA1 value of the file;

[0029] Step S200: dividing the total size of the file by 10M, and rounding up the result, to calculate the total number of file slices;

[0030] Step S300: the bastion machine server queries the database by using the parameters of the file in step S100 and step S200, if the same file is not matched in the database, the file is written in the database, and the uploading state field of the file is set to uploading;

[0031] Step S400: The front end uses the slice method of the Blob object to slice the file according to every 10M size, records the serial number of each slice, calculates the SHA1 value of the slice, and calculates the MD5 of the slice by taking the SHA1 value of the slice, the file name and the serial number as salt values.

[0032] Step S500: The bastion host server sequentially queries the server Caffeine cache, Redis cache and database according to the parameters of the slice in step S400, and if none of them matches the slice, it means that the slice has not been uploaded, so the slice data is written into the Caffeine cache, and the upload state of the slice is set to uploading.

[0033] Step S600: The front end calls the upload interface to upload the slice MD5, slice serial number, slice content and file name to the bastion host server, the bastion host server accepts the file slice and stores it in the Caffeine cache, Redis cache and database, after successful upload, calculates the SHA1 value of the slice, and calculates the MD5 of the slice by taking the SHA1 value of the slice, the file name and the serial number of the slice as salt values; if the MD5 value of the slice transmitted by the front end is consistent, it means that the content of the slice is uploaded correctly, and the upload state of the slice in the Caffeine cache, Redis cache and database is changed to upload success.

[0034] Step S700: The front end calls the file merging interface to upload the file name and file SHA1 value as parameters to the bastion host server, the bastion host server sequentially queries the Caffeine cache, Redis cache and database to find all the slices of the file that have been uploaded, performs file slice merging, calculates the MD5 value of the file after successful merging, and if it is consistent with the MD5 value of the upload parameter, the file upload is successful and a message is sent to the front end, all slice caches are deleted, and the upload state of the file in the database is changed to upload success.

[0035] Preferably, in step S300, the file check interface is called to upload the file name, file size, total number of file slices and file SHA1 value to the bastion host server, and the bastion host server queries the database according to the four parameters.

[0036] Preferably, in step S300, if the database matches the same file, the upload state of the file is checked; if the upload state is upload failure, it means that the file upload record already exists, but the upload is not completed, so step S400 is continued to be executed for reuploading; if the upload state is uploaded, it means that the file has been successfully uploaded to the server, so the server directly sends a file uploaded successfully message to the client.

[0037] Preferably, the step S500 comprises: the front end sequentially calls the file fragment checking interface according to the order of the sequence number of the fragments, uploads the MD5 of each fragment, the sequence number of the fragment and the file name to the bastion host server, and the bastion host server sequentially queries the server Caffeine cache, Redis cache and database according to the three parameters.

[0038] Preferably, in the step S500, if the same fragment is matched in any one of the Caffeine cache, the Redis cache and the database, the uploading state of the fragment is checked; if the uploading state is uploading failure, it is indicated that the file uploading record exists but the uploading is not completed, so the fragment needs to be re-uploaded, and the step S600 is executed; if the uploading state is uploading success, it is indicated that the fragment has been successfully uploaded to the server, if this is the last fragment, the step S700 is executed; otherwise, the step S500 is executed to check the next fragment.

[0039] Preferably, in the step S600, if the server detects network disconnection or calculates that the MD5 value is inconsistent with the uploading parameter MD5 value, the fragment uploading fails, the uploading state of the fragment in the Caffeine cache, the Redis cache and the database is changed to uploading failure, and the uploading state of the file is changed to uploading failure.

[0040] The application firstly uses the SHA1 algorithm to judge the uniqueness of the file, the collision probability of the SHA1 algorithm is less than that of the MD5, and the SHA1 algorithm is a safer and more reliable algorithm; secondly, the application uses MD5 with salt, uses the SHA1 value of the fragment, the file name and the sequence number of the fragment as the salt value, and calculates the MD5 of each fragment, which has three advantages:

[0041] (4) the number of large file fragments is large, and the performance is higher when using MD5 calculation;

[0042] (5) using the three values as part of the salt value is equivalent to calculating the accuracy of SHA1 and the accuracy of MD5, and the accuracy of the uniqueness of the file fragment is further improved;

[0043] (6) using SHA1 as the salt value to calculate MD5 can greatly improve the security of MD5, and the possibility of MD5 being parsed and collided is greatly reduced.

[0044] When the file is uploaded, the front end does not know the uploading state of the file and each slice of the file, and whether the file and the uploading state of the file are queried from the database according to step S300. If the file has been uploaded, the file does not need to be uploaded again. If the file does not exist or the file fails to be uploaded, the uploading state of each slice is checked according to step S500 in sequence from the Caffeine cache, the Redis cache and the database value. If the uploading state of the slice is uploaded, it is indicated that the slice of the file has been successfully uploaded before, but other slices may not be successfully uploaded or the file slices fail to be combined, so that the whole file fails to be uploaded. Then, the file uploading this time can skip the slice, so that the uploading time can be saved, and the effect of breakpoint resume is achieved.

[0045] When the slice is saved, the two-level cache of the Caffeine cache + the Redis cache is used, the Caffeine cache is read and written first, the Caffeine cache is a local cache compared with the Redis cache, and the reading and writing speed is faster and the efficiency is higher. If the Caffeine cache is not matched, the Redis cache is queried. Finally, the database is queried if necessary. In this way, the reading and writing times of the database are greatly reduced, the reading and writing times of the Redis cache are reduced, and the storage efficiency of the server is improved.

[0046] Embodiment 2

[0047] A file slice uploading method based on a bastion host operation and maintenance, the file is uploaded to a hosting host by using the bastion host to perform host operation and maintenance through slice uploading, and the steps are as follows:

[0048] 1. A user clicks an uploading button on an H5 operation and maintenance page, selects a file to be uploaded, and clicks a confirm button.

[0049] 2. A front end reads the file name and the file size of the uploading file by using a Blob object of Javascript, and calculates a SHA1 value of the file.

[0050] 3. The total size of the file is divided by 10M, and the result is rounded up to calculate the number of all slices. (This step only calculates the number of slices, and does not perform file slicing.)

[0051] 4. A file checking interface is called, the file name, the file size, the total number of file slices and the file SHA1 are uploaded to a bastion host server, and the server queries a database according to the four parameters.

[0052] a) If the same file is not matched in the database, it is indicated that the file has not been uploaded, the file data is written in the database, the file uploading state field is set to uploading, and step 5 is continued.

[0053] b) If the database matches the same file, check the upload status of the file

[0054] i. If the upload status is upload failed, it means that the file upload record already exists, but the upload may not be completed due to network disconnection or other reasons, continue to step 5 to reupload;

[0055] ii. If the upload status is uploaded, it means that the file has been successfully uploaded to the server, the server directly sends a file upload success message to the client, and the process ends.

[0056] 5、 The front end uses the slice method of the Blob object to slice the file according to every 10M size, records the sequence number (index) of each slice, calculates the SHA1 value of the slice, and calculates the MD5 of the slice using the slice SHA1 value, the file name and the sequence number of the slice as the salt value. (This step performs file slicing)

[0057] 6、 The front end sequentially calls the file slice check interface according to the slice sequence number to upload the MD5 of each slice, the slice sequence number and the file name to the bastion host server. The server queries the server Caffeine cache according to these three parameters:

[0058] a) If the Caffeine cache does not match the same slice, Redis cache is searched again. If the Redis cache also does not match successfully, the database is queried. If the database also does not match the slice, it means that the slice has not been uploaded, the slice data is written in the Caffeine cache, and the slice upload status is set to uploading, and step 7 is continued;

[0059] b) If the Caffeine cache or Redis cache or database matches the same slice, check the upload status of the slice

[0060] i. If the upload status is upload failed, it means that the file upload record already exists, but the upload may not be completed due to network disconnection or other reasons, the slice needs to be reuploaded, and step 7 is executed;

[0061] ii. If the upload status is uploaded, it means that the slice has been successfully uploaded to the server. If this is the last slice (slice sequence number = total number of slices), step 8 is executed. If this is not the last slice, step 6 is repeated to check the next slice;

[0062] 7. The front-end calls the file fragment upload interface, uploads the fragment MD5, fragment sequence number, fragment content and file name to the bastion server, the server accepts the file fragment, stores the fragment into Caffeine cache and Redis cache and database, after successful upload, calculates the fragment SHA1 value, then uses the fragment SHA1 value, file name and sequence number of the fragment as salt value to calculate the MD5 of the fragment:

[0063] a) If the MD5 value is consistent with the MD5 value transmitted by the front-end, it means that the content of the fragment is uploaded correctly, change the upload state of the fragment in Caffeine cache, Redis cache and database to upload success

[0064] i. If it is the last fragment (fragment sequence number = total number of fragments), execute step 8;

[0065] ii. If it is not the last fragment, repeat step 6 to check the next fragment;

[0066] b) If the server detects network disconnection or calculates that the MD5 value is inconsistent with the MD5 value of the upload parameter, the fragment upload fails, changes the upload state of the fragment in Caffeine cache, Redis cache and database to upload failure, changes the upload state of the file to upload failure, and the process ends;

[0067] 8. The front-end calls the file merging interface, uploads the file name and file SHA1 value to the bastion server as parameters, the server queries Caffeine cache and Redis cache and database to find all uploaded fragments belonging to the file in sequence, performs file fragment merging, and calculates the file SHA1 value after successful merging

[0068] a) If the MD5 value is consistent with the MD5 value of the upload parameter, the file upload is successful and a message is sent to the front-end, all fragment caches are deleted, the upload state of the file in the database is changed to upload success, and the process ends

[0069] b) If the MD5 value is inconsistent with the MD5 value of the upload parameter, the file upload is incorrect, the upload state of the file is changed to upload failure, and the process ends.

[0070] The ordinary fragment upload calculates the MD5 value of the whole file and each fragment as a unique identifier to judge the uniqueness of the file and the fragment, but the MD5 algorithm has been proved to have a possibility of repetition when the data volume is large, which may affect the uniqueness judgment. The application first uses SHA1 algorithm to judge the file uniqueness, the collision probability of SHA1 algorithm is less than MD5, which is a safer and more reliable algorithm; secondly, the application uses MD5 with salt, uses the fragment SHA1 value, file name and sequence number of the fragment as salt value to calculate the MD5 of each fragment, which has three advantages:

[0071] ① Large file fragment quantity, use MD5 calculation, higher performance;

[0072] ② Use the three values as part of the salt value, which is equal to calculating the accuracy of SHA1 and the accuracy of MD5, and further improving the accuracy of file fragment uniqueness;

[0073] ③ Using SHA1 to calculate MD5 can greatly improve the security of MD5, and the possibility of MD5 being parsed and collided is greatly reduced.

[0074] When uploading a file, the front end does not know the upload status of the file and each fragment of the file. According to the implementation step 4 of the application, it is inquired whether the file and the file upload state exist in the database. If the file has been uploaded, it does not need to be uploaded again. If there is no file or the file upload fails, according to the implementation step 6 of the application, the caffeine cache, the Redis cache and the database value are inquired in sequence to check the upload state of each fragment. If the fragment upload state is uploaded, it indicates that the fragment of the file has been successfully uploaded before, but other fragments may not be successfully uploaded or the file fragment merging fails, resulting in the whole file upload failure. Therefore, the file upload can skip the fragment, which can save the upload time and achieve the effect of breakpoint resume. When saving the fragment, the two-level cache of caffeine cache + Redis cache is used. First, read and write caffeine cache for reading and writing. Compared with Redis cache, caffeine cache is local cache, and the reading and writing speed is faster and the efficiency is higher. If caffeine cache is not matched, Redis cache is inquired. Finally, if necessary, the database is inquired. In this way, not only the number of database read and write times is greatly reduced, but also the number of Redis cache read and write times is reduced, and the storage efficiency of the server is improved.

[0075] The above is only the preferred embodiment of the application, and does not limit the application in any form. Any simple modification or equivalent change based on the technical essence of the application to the above embodiment falls within the protection scope of the application.

Claims

1. A file chunk uploading method based on bastion operation and maintenance, characterized in that, The method comprises the following steps: Step S100: uploading a file and reading the file name and file size of the file, and calculating the SHA1 value of the file; Step S200: dividing the total size of the file by the set size, and rounding up the result to calculate the total number of file fragments; Step S300: the bastion host server queries the database by using the parameters of the file in steps S100 and S200, if the database does not match the same file, the file is written in the database, and the file upload state field is set to uploading; Step S400: the front end divides the file into fragments according to the set size, records the serial number of each fragment, calculates the SHA1 value of the fragment, and calculates the MD5 of the fragment by taking the SHA1 value, the file name and the serial number of the fragment as the salt value; Step S500: the bastion host server sequentially queries the server Caffeine cache, Redis cache and database according to the parameters of the fragment in step S400, if none of them matches the fragment, it means that the fragment has not been uploaded, then the fragment data is written in the Caffeine cache, and the upload state of the fragment is set to uploading; Step S600: the front end calls the upload interface, uploads the fragment MD5, fragment serial number, fragment content and file name to the bastion host server, the bastion host server accepts the file fragment, and stores the fragment into the Caffeine cache, Redis cache and database, after successful uploading, the SHA1 value of the fragment is calculated, and then the SHA1 value of the fragment, the file name and the serial number of the fragment are taken as the salt value to calculate the MD5 of the fragment; If the MD5 value of the fragment transmitted by the front end is consistent, it means that the content of the fragment is uploaded correctly, and the upload state of the fragment in the Caffeine cache, Redis cache and database is changed to uploading success; Step S700: the front end calls the file merging interface, uploads the file name and file SHA1 value to the bastion host server as parameters, the bastion host server sequentially queries the Caffeine cache, Redis cache and database, finds all the fragments of the file that have been uploaded, performs file fragment merging, calculates the MD5 value of the file after successful merging, if the MD5 value is consistent with the upload parameter, the file is successfully uploaded and a message is sent to the front end, all fragment caches are deleted, and the upload state of the file in the database is changed to uploading success.

2. The file slice uploading method based on the bastion machine operation and maintenance according to claim 1, characterized in that, In step S300, a file checking interface is called, the file name, file size, total number of file fragments and file SHA1 value are uploaded to the bastion host server, and the bastion host server queries the database according to the four parameters.

3. The file chunk uploading method based on the bastion machine operation and maintenance according to claim 2, characterized in that, In step S300, if the database matches the same file, the upload state of the file is checked; if the upload state is uploading failure, it means that the file upload record already exists, but the uploading is not completed, then step S400 is continued to be executed for reuploading; if the upload state is uploaded, it means that the file has been successfully uploaded to the server, and the server directly sends a file uploaded successfully message to the client.

4. The file slice uploading method based on the bastion machine operation and maintenance according to claim 1, characterized in that, The step S500 comprises: the front end sequentially calls the file fragment checking interface according to the sequence number of the fragments, uploads the MD5 of each fragment, the sequence number of the fragment and the file name to the bastion server, and the bastion server sequentially queries the server Caffeine cache, Redis cache and database according to the three parameters.

5. The file slice uploading method based on the bastion machine operation and maintenance according to claim 4, characterized in that, In the step S500, if the same fragment is matched in any one of the Caffeine cache, the Redis cache and the database, the uploading state of the fragment is checked; if the uploading state is uploading failure, it is indicated that the file uploading record exists but the uploading is not completed, so the fragment needs to be re-uploaded, and the step S600 is executed; if the uploading state is uploading success, it is indicated that the fragment has been successfully uploaded to the server, if this is the last fragment, the step S700 is executed; otherwise, the step S500 is executed to check the next fragment.

6. The file slice uploading method based on the bastion machine operation and maintenance according to claim 1, characterized in that, In the step S600, if the server detects network disconnection or calculates that the MD5 value is inconsistent with the uploading parameter MD5 value, the fragment uploading fails, the uploading state of the fragment in the Caffeine cache, the Redis cache and the database is changed to uploading failure, and the uploading state of the file is changed to uploading failure.

7. The file slice uploading method based on the bastion machine operation and maintenance according to claim 1, characterized in that, In the step S100, the file name and the file size of the file are read by using the Blob object of Javascript; and in the step S400, the file is fragmented by using the slice method of the Blob object.