Failure file cleaning method and device, computer device and storage medium

Abstract

The application relates to a failure file cleaning method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a file cleaning instruction, determining a to-be-cleaned file in a file server; through a file selection interface of each business data table, obtaining business file identifiers associated with each business data, and determining the business file identifiers associated with each business data as valid file identifiers; the business data table is used for storing the corresponding relationship between each business data and each business file; according to the valid file identifiers, the to-be-cleaned files are matched, and the to-be-cleaned files that fail to match the valid file identifiers are determined as failure files; based on the failure files, a failure file cleaning instruction is generated, and the file server is controlled to clean the failure files. By adopting the method, the failure files can be accurately cleaned, and the storage space is saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a method and device for cleaning invalid files, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] A business file server is a server for storing business files. When a business file is successfully uploaded to the business file server, the business data of each application scenario will be associated with the corresponding business file in the file server.

[0003] With the gradual increase of business services, the amount of business files received and stored in the business file server also increases. Since the business files are usually associated with the business data of one or more business applications, the business file server cannot accurately determine the business files that need to be cleaned, resulting in a large number of invalid files being retained in the business database, causing waste of storage space. SUMMARY

[0004] Therefore, it is necessary to provide a method and device for cleaning invalid files, a computer device, a computer readable storage medium and a computer program product, which can accurately clean invalid files and save storage space.

[0005] In a first aspect, the present application provides a method for cleaning invalid files, the method comprising:

[0006] In response to a file cleaning instruction, determining a to-be-cleaned file in a file server;

[0007] Through a file selection interface of each business data table, obtaining a business file identifier associated with each business data, and determining the business file identifier associated with each business data as a valid file identifier; the business data table is used to store the correspondence between each business data and each business file;

[0008] According to the valid file identifier, matching the to-be-cleaned file, and determining a to-be-cleaned file that fails to match the valid file identifier as an invalid file;

[0009] Based on the invalid file, generating an invalid file cleaning instruction, and controlling the file server to clean the invalid file.

[0010] In one embodiment, determining a to-be-cleaned file in a file server comprises:

[0011] Obtaining the modification time of each business file stored in the file server;

[0012] According to the modification time and the current time, determining the modification interval of each business file;

[0013] determine the service file whose modification interval duration exceeds the preset duration threshold as a file to be cleaned up.

[0014] In one of the embodiments, the service file identifier associated with each service data is obtained through the file selection interface of each service data table, including:

[0015] The file selection interface pre-set in the data access layer of each service data table is called to obtain the service file identifier associated with each service data; the service file identifier is obtained by each service corresponding to the file selection interface, and the effective file selection of the service data table is performed.

[0016] In one of the embodiments, the file selection interface pre-set in the data access layer of each service data table is called to obtain the service file identifier associated with each service data, including:

[0017] The data access layer of each service data table is obtained, and the data access layer is pre-set with the file selection interface;

[0018] According to the service to which each data access layer belongs, a processing thread is allocated to each data access layer;

[0019] Based on each processing thread, the file selection interface of each data access layer is called to instruct each service corresponding to each data access layer to implement the file selection interface, and the effective file selection of the service data table is performed to obtain the service file identifier associated with each service data.

[0020] In one of the embodiments, the effective file identifier is matched with the file to be cleaned up, and the file to be cleaned up that fails to match the effective file identifier is determined as an invalid file, including:

[0021] Each effective file identifier obtained by each processing thread is matched with the file identifier of each file to be cleaned up through each processing thread;

[0022] An effective use identifier is set for the file to be cleaned up that matches successfully in each processing thread;

[0023] After the matching of each processing thread is completed, the file to be cleaned up that does not have the effective use identifier is determined as an invalid file.

[0024] In one of the embodiments, the file selection interface pre-set in the data access layer of each service data table is called to obtain the service file identifier associated with each service data, including:

[0025] The file selection interface preset in the data access layer of each business data table is called in batches according to a preset calling quantity using a stream query database framework;

[0026] The batch business file identifiers fed back by the file selection interface corresponding to the business implementation to which each batch data access layer belongs are received;

[0027] The batch business file identifiers are collected to obtain business file identifiers associated with each business data.

[0028] In a second aspect, the application further provides a device for cleaning invalid files, and the device comprises:

[0029] An instruction response module is configured to determine files to be cleaned in a file server in response to a file cleaning instruction;

[0030] An effective file determination module is configured to acquire business file identifiers associated with each business data through a file selection interface of each business data table, and determine the business file identifiers associated with each business data as effective file identifiers; the business data table is configured to store a corresponding relationship between each business data and each business file;

[0031] An invalid file determination module is configured to match the files to be cleaned according to the effective file identifiers, and determine the files to be cleaned that fail to match the effective file identifiers as invalid files;

[0032] A cleaning module is configured to generate an invalid file cleaning instruction based on the invalid files, and control the file server to clean the invalid files.

[0033] In a third aspect, the application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0034] In a fourth aspect, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0035] In a fifth aspect, the application further provides a computer program product comprising a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0036] The aforementioned expired file cleanup method, apparatus, computer equipment, storage medium, and computer program product, in response to a file cleanup command, after identifying the files to be cleaned in the file server, automatically obtains the business file identifier associated with each business data based on the correspondence between each business data and each business file stored in the business data tables through a pre-set file selection interface for each business data table. If a business file is associated with each business data, it can be determined as a valid file, and the business file identifier associated with each business data is determined as a valid file identifier. The files to be cleaned are then matched against the valid file identifier; files that do not match the valid file identifier are determined as expired files. By using the valid file identifier for secondary filtering of the files to be cleaned, the accuracy of expired file identification can be greatly improved. Subsequently, an expired file cleanup command is generated based on the accurate expired files, controlling the file server to clean up the expired files, effectively saving storage space on the file server. Attached Figure Description

[0037] Figure 1 This is a diagram illustrating the application environment of a failed file cleanup method in one embodiment.

[0038] Figure 2 This is a flowchart illustrating a method for cleaning up expired files in one embodiment;

[0039] Figure 3 This is a flowchart illustrating the steps for determining files to be cleaned in a file server in one embodiment.

[0040] Figure 4 This is a flowchart illustrating the steps of calling the file selection interface pre-set in the data access layer of each business data table to obtain the business file identifier associated with each business data in one embodiment.

[0041] Figure 5 This is a flowchart illustrating the steps of matching files to be cleaned based on valid file identifiers in one embodiment, and identifying files to be cleaned that do not match the valid file identifiers as invalid files.

[0042] Figure 6 This is a flowchart illustrating the steps of calling the file selection interface pre-set in the data access layer of each business data table to obtain the business file identifier associated with each business data in one embodiment.

[0043] Figure 7 This is a diagram illustrating the application environment of the expired file cleanup method in another embodiment;

[0044] Figure 8 This is a flowchart illustrating the method for cleaning up expired files in another embodiment;

[0045] Figure 9A structural block diagram of the invalid file cleaning device in one embodiment;

[0046] Figure 10 An internal structural diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] The invalid file cleaning method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 . The file cleaning component 102 communicates with the file server 104 and each service database 106 through a network respectively. The data storage system can store data required to be processed by the file cleaning component 102. The data storage system can be integrated on the file cleaning component 102, or placed on a cloud or other network server. The file cleaning component 102 determines the to-be-cleaned files in the file server 104 in response to a file cleaning instruction, acquires the service file identifiers associated with each service data through a file selection interface of each service data table stored in each service database 106, determines the service file identifiers associated with each service data as valid file identifiers, matches the to-be-cleaned files according to the valid file identifiers, determines the to-be-cleaned files that fail to match the valid file identifiers as invalid files, generates an invalid file cleaning instruction based on the invalid files, and controls the file server 104 to clean the internal invalid files. The file cleaning component 102 can be integrated on the file server 104 or on a cloud server. The service database 106 and the file server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0049] In one of the embodiments, the file server can be a MongoDB file server.

[0050] In one embodiment, as shown in Figure 2 , an invalid file cleaning method is provided. Taking the file cleaning component in Figure 1 as an example, the method comprises the following steps:

[0051] Step 202, in response to a file cleaning instruction, determining to-be-cleaned files in a file server.

[0052] The file cleaning instruction is an instruction for instructing to clean the invalid files stored in the file server. The file cleaning instruction can be generated by a user based on a user terminal and sent to the file cleaning component, or generated by the file cleaning component at a preset time interval. For example, a designer can pre-set a cleaning period, and the file cleaning component generates the file cleaning instruction at each cleaning period to clean the invalid files stored in the file server.

[0053] The file server is a server for storing service files. After the service files are successfully uploaded to the service file server, the file server generates a unique file identifier of the service file and returns the file identifier to the corresponding service. The service stores the file identifier and the corresponding service data in the service database used by the service. It should be noted that the service file can be considered as a file separated from the service data and stored independently. For example, the service file includes a portrait picture related to personal information, an electronic certificate, and the like. The service file generally includes a file name, a file size, a file type, a date, a file byte code, and the like. The service file is generally large, and separating the service file from the service data can effectively reduce the burden of the service database.

[0054] The to-be-cleaned file is a service file with a possibility of invalidity, which is screened from the file server according to a preset investigation rule. After receiving the file cleaning instruction, the file cleaning component investigates all service files in the file server according to the preset investigation rule, and determines the service file with the possibility of invalidity as the to-be-cleaned file in the file server.

[0055] Specifically, the file cleaning component responds to the file cleaning instruction, investigates the file server in association, and determines the to-be-cleaned file in the file server.

[0056] In step 204, the file selection interface of each service data table is used to obtain the service file identifier associated with each service data, and the service file identifier associated with each service data is determined as a valid file identifier. The service data table is used to store the correspondence between each service data and each service file.

[0057] The file selection interface is a general method interface pre-configured for each service data table. The file selection interface corresponds to each service data table in one-to-one correspondence. The file cleaning component can obtain the service file identifier associated with each service data through the file selection interface corresponding to each service data table.

[0058] The business data table is a relationship mapping table for storing the correspondence between each business data and each business file. The business data table is arranged in the business database. The business data of each scenario of the application service is stored in the business database. The business database sets a corresponding business data table for each scenario of the business, and is used to store the mapping relationship between the business data under the business and the business file stored in the file server. It can be understood that the business data table associates each business file with each business data according to the unique identifier of each business file.

[0059] Specifically, the file cleaning component obtains the business file identifiers associated with each business data according to the correspondence between each business data and each business file stored in the business data table through the file selection interface of each business data table. Since the obtained business file identifiers are all unique identifiers of the business files associated with each business data, it can be considered that the business file corresponding to the identifier is a valid file. The obtained business file identifier is determined as a valid file identifier.

[0060] Step 206, according to the valid file identifier, the to-be-cleaned file is matched, and the to-be-cleaned file which fails to match the valid file identifier is determined as an invalid file.

[0061] Specifically, according to the valid file identifier, the to-be-cleaned file is matched. If the to-be-cleaned file matches the valid file identifier successfully, it means that the to-be-cleaned file has an association relationship with the business data, and is a valid file and cannot be deleted. If the to-be-cleaned file fails to match the valid file identifier, it means that the to-be-cleaned file does not have an association relationship with each business data, and can be deleted. Therefore, the to-be-cleaned file which fails to match the valid file identifier is determined as an invalid file.

[0062] Step 208, generating an invalid file cleaning instruction based on the invalid file, and controlling the file server to clean the invalid file.

[0063] Specifically, the file cleaning component generates an invalid file cleaning instruction based on the file identifier of the invalid file, and sends the invalid file cleaning instruction to the file server. The file server responds to the invalid file cleaning instruction, and cleans the invalid file stored in the file server according to the file identifier of the invalid file carried in the file cleaning instruction.

[0064] In one of the embodiments, if the file server is MongoDB, the file server will clean the GridFS file corresponding to the invalid file when cleaning the invalid file.

[0065] In the above invalid file cleaning method, in response to the file cleaning instruction, after determining the to-be-cleaned files in the file server, the file selection interface of each business data table is used to obtain the business file identifiers associated with each business data according to the correspondence between each business data and each business file stored in the business data table. If the business file is associated with each business data, the business file is determined to be a valid file, the business file identifiers associated with each business data are determined to be valid file identifiers, and the to-be-cleaned files are matched according to the valid file identifiers. The to-be-cleaned files that fail to match the valid file identifiers are determined to be invalid files. The to-be-cleaned files are screened again using the valid file identifiers, which greatly improves the accuracy of invalid file determination. Then, the invalid file cleaning instruction is generated according to the accurate invalid files, and the file server is controlled to clean the invalid files, thereby effectively saving the storage space of the file server.

[0066] During the business, a certain processing time is required to establish the association between the corresponding business file and the business data. Based on this, in one embodiment, as shown in Figure 3 The determination of the to-be-cleaned files in the file server includes:

[0067] In step 302, the modification time of each business file stored in the file server is obtained.

[0068] Specifically, the file cleaning component obtains the modification time of each business file stored in the file server. The modification time of the business file is a time parameter that is updated after each modification of the file content in the business file. It can be considered that the obtained modification time of the business file is the modification time closest to the time distance initiated by the file cleaning component.

[0069] In step 304, the modification interval length of each business file is determined according to the modification time and the current time.

[0070] Specifically, the file cleaning component determines the modification interval length of each business file by calculating the time difference between the current time and the obtained modification time of each business file.

[0071] In step 306, the business file with a modification interval length greater than a preset time threshold is determined to be a to-be-cleaned file.

[0072] The preset time length threshold is a time length judgment parameter used to determine whether the service file is a pre-associated file. The pre-associated file refers to a service file that is being uploaded in a service and has not been associated with service data. After the subsequent service is completed, the service file will be associated with the corresponding service data. It can be understood that the preset time length threshold can be determined according to the time length required for the service file to be associated with the service data. For example, if the average time length required for the service file to be associated with the service data is 24 hours, the preset time length threshold is set to 24 hours.

[0073] Specifically, the file cleaning component compares the modification interval time length with the preset time length threshold. If the modification interval time length of the service file does not exceed the preset time length threshold, it means that the service file has a certain possibility of being a pre-associated file. If the pre-associated file is determined as a file to be cleaned, the subsequent deletion will cause a mistaken deletion operation, resulting in a connection disorder between the file server and the service. Therefore, the file cleaning component only determines the service file with the modification interval time length exceeding the preset time length threshold as the file to be cleaned, and performs subsequent invalid file screening.

[0074] In this embodiment, by setting the preset time length threshold, the service files in the file server are investigated, and the service files with the modification interval time length not exceeding the preset time length threshold are excluded from the files to be cleaned. This can effectively avoid the situation that the mistaken deletion of the pre-associated file causes a connection disorder between the file server and the services, and improves the judgment accuracy of the files to be cleaned, thereby providing a data basis for the accurate judgment of the subsequent invalid files.

[0075] In actual use, because the application scenarios of the services are different and the service processing logics are different, the fields of the associated file identifiers of the service data tables corresponding to the services can be different. For example, the associated file identifier field of the personal avatar can be imageId, and the associated file identifier field of the product picture in the product service scenario can be pictureId. The valid file judgment logics in the services can also be different. For example, the personal information avatar is updated, and the imageId is updated. The imageId in the personal information table is valid. For the product deletion, only the record is soft deleted, and the record state is marked as deleted. The valid pictureId needs to exclude the deleted product record.

[0076] In order to realize the scalability of the invalid file cleaning method, in one embodiment, the file selection interface of each service data table is used to obtain the service file identifier associated with each service data, including:

[0077] The file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifiers associated with each business data; the business file identifiers are obtained by each business to which the data access layer belongs respectively implementing the corresponding file selection interface, and effective file selection is performed on the business data table.

[0078] The data access layer, namely the Mapper or Dao layer, can perform data persistence operation on the database, and its method statement is directly aimed at database operation, and mainly implements some adding, deleting, modifying and inquiring operation. The designer pre-sets the file selection interface in the Mapper or Dao layer corresponding to each business data table, and the file selection (selectFileIds) method of the common interface CommonFileDao is implemented in the Mapper or Dao layer through the file selection interface. The function of the interface method is to obtain the effective business file identifiers in the related business data table.

[0079] Specifically, the file cleaning component calls the file selection interface pre-set in the data access layer of each business data table to obtain the business file identifiers associated with each business data, each business to which the data access layer belongs respectively implements the file selection method in the file selection interface, and effective file selection is performed on the business data table through the effective file judgment logic of each business itself. According to the corresponding relationship stored in each business data table, the business file identifiers associated with each business data are determined, and the business file identifiers are fed back to the file cleaning component through the file selection interface. The file cleaning component receives the business file identifiers associated with each business data.

[0080] In the embodiment, by providing a unified and common file selection interface, and by each business self-implementing the corresponding effective file judgment logic, the problem that the effective file judgment logics of each business data are different and the fields of the associated file identifiers are different can be effectively solved, so that the invalid file cleaning method can be extended to multiple business data.

[0081] In one embodiment, as Figure 4 The file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifiers associated with each business data, including:

[0082] In step 402, the data access layer of each business data table is obtained, and the data access layer is pre-provided with the file selection interface.

[0083] Specifically, the file cleaning component obtains the data access layer, namely the Mapper or Dao layer, which implements the CommonFileDao interface through the Spring context ApplicationContext.

[0084] In step 404, a processing thread is allocated to each data access layer according to the business to which the data access layer belongs.

[0085] Specifically, the file cleaning component determines the business to which each data access layer belongs, and allocates a corresponding processing thread to each data access layer according to the business to which each data access layer belongs.

[0086] In step 406, each data access layer is called by each processing thread based on a file selection interface, and each data access layer is instructed to implement the file selection interface by the business to which the data access layer belongs, to perform effective file selection on the business data table, and obtain the business file identifier associated with each business data.

[0087] Specifically, the file cleaning component calls the file selection interface of each data access layer through the processing thread corresponding to each data access layer in the thread pool, instructs the business to which the data access layer belongs to implement the file selection method in the file selection interface, and performs effective file selection on the business data table through the effective file judgment logic of the business itself. According to the corresponding relationship stored in the business data table, the business file identifier associated with each business data is determined, and the business file identifier associated with each business data is fed back to the file cleaning component, and the file cleaning component obtains the business file identifier associated with each business data.

[0088] In this embodiment, by configuring a corresponding processing thread for each data access layer, the parallel processing amount is improved by using a multi-thread parallel processing mode, which can effectively improve the running efficiency of the component.

[0089] Further, as shown in Figure 5 , the matching of the to-be-cleaned files with the valid file identifiers is performed, and the to-be-cleaned files that fail to match the valid file identifiers are determined as invalid files, including:

[0090] In step 502, each processing thread matches the valid file identifier obtained by each processing thread with the file identifier of each to-be-cleaned file.

[0091] Specifically, the file cleaning component matches the valid file identifier obtained by each processing thread with the file identifier of each to-be-cleaned file through the processing thread corresponding to each data access layer.

[0092] In step 504, an effective use identifier is set for the to-be-cleaned file that matches successfully in each processing thread.

[0093] The use identifier is an identifier information used to mark the use state of the to-be-cleaned file, and the effective use identifier represents that the business file to which the identifier is set is currently in an associated state and has an association relationship with the business data.

[0094] Specifically, in order to more intuitively display the use state of each to-be-cleaned file, each thread sets an effective use identifier for the to-be-cleaned file that matches successfully after the matching is completed.

[0095] Step 506, after each processing thread is matched, the file to be cleaned without a valid use identifier is determined as an invalid file.

[0096] Specifically, after each processing thread in the thread pool is matched, if the to-be-cleaned file does not have a valid use identifier, it can be considered that the to-be-cleaned file has been invalidated and does not have an association with each business data. The file cleaning component determines the to-be-cleaned file without a valid use identifier in the file database as an invalid file.

[0097] In one embodiment, in order to more directly determine the invalid file, the file cleaning component sets an invalid use identifier for each to-be-cleaned file obtained by investigating the file server. The invalid use identifier represents that the business file setting the identifier is currently in a to-be-cleaned state. When each thread matches the valid file identifier with each to-be-cleaned file, if the matching is successful, the use identifier of the to-be-cleaned file matched successfully is set as a valid use identifier. When all threads in the thread pool are matched, the to-be-cleaned file with a use identifier still being an invalid use identifier is determined as an invalid file.

[0098] In the above embodiment, by adding a use identifier to the to-be-cleaned file, the invalid file identification problem is solved, and the determination process of the invalid file becomes more intuitive and accurate.

[0099] In addition to the multi-thread parallel scheme, in order to further improve the running efficiency, in one embodiment, as shown in Figure 6 , a file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifier associated with each business data, including:

[0100] Step 602, using a streaming query database framework, the file selection interface pre-set in the data access layer of each business data table is called in batches according to a preset calling quantity.

[0101] Among them, the streaming query database framework (Mybatis) is a Java ORM framework that can realize streaming query, and can realize data streaming output through ResultHandler and StreamingStatementHandler.

[0102] Specifically, the file processing component uses a stream query database framework in the file selection interface, and calls the file selection interface pre-configured in the data access layer of each business data table in batches according to a preset number of calls. The preset number of calls is determined according to the actual processing performance of the server. If the actual processing performance of the server determines that 3 file selection interfaces can be called in each batch, and there are currently 9 file selection interfaces, the file selection interfaces will be divided into 3 batches, and 3 file selection interfaces will be called in each batch to obtain the business file identifiers associated with each business data.

[0103] Step 604, receiving the batch business file identifiers fed back by the file selection interface corresponding to the business implementation to which the data access layer belongs.

[0104] Specifically, after the file cleaning component calls the file selection interface in batches, the data access layer configured with the file selection interface in the batch determines the business to which the data access layer belongs. Each business will implement the file selection method in the corresponding file selection interface according to its own effective file selection logic, and feed back the business file identifiers associated with the corresponding business data. The file cleaning component obtains the batch of corresponding allocated business file identifiers.

[0105] Step 606, collecting each batch of business file identifiers to obtain the business file identifiers associated with each business data.

[0106] Specifically, after the stream query is completed, the file cleaning component collects the batch of business file identifiers obtained to obtain the business file identifiers associated with each business data.

[0107] In the above embodiment, the stream query database framework is used in the file selection interface to perform stream query operation, without taking out all data at once, and can be processed or taken out in batches. One query specifies the number of data, until all data is processed, which can avoid the abnormal situation of insufficient memory caused by too large business data, and save the consumption of memory.

[0108] MongoDB is a database that can store binary data of BSON structure, and can be used as a file server. After successfully uploading a file to MongoDB, MongoDB will generate a unique file id (fileId) for the file. The business data of each scenario of the application service is associated with the corresponding business file by storing the file id. The following scenarios will often produce invalid files for the file server:

[0109] Scenario one, after uploading a file, there is no business data association.

[0110] Scenario two, business data is deleted, and the associated file is not deleted from the file server.

[0111] Scenario three, the file is associated with multiple business data, when the business data is deleted, the file needs to be deleted later.

[0112] The above scenarios often occur in business, which will retain a large number of invalid files in the MongoDB file server, causing waste of storage space. In order to save storage space, it is necessary to clean up the invalid files regularly. However, since some invalid files have no record of file id in the business data table, the invalid files cannot be directly identified; and the associated file id fields of each business data table may be different, and the judgment logic of each business data valid file is also different. Therefore, the MongoDB file server cannot accurately clean up the invalid files.

[0113] Therefore, in one embodiment, a method for cleaning up invalid files is provided, which is applied in Figure 7 The application environment in which the method is applied is taken as an example for illustration, in which each business is connected with a database and a MongoDB file server, and each business uploads the business file generated during operation to the MongoDB file server and stores the business data in the corresponding database. After the business file is successfully uploaded, the MongoDB file server generates a unique file identifier of the business file and returns the file identifier to the corresponding business. The business uploads the business file identifier together with the business data when uploading the business data to the business database, and the business database generates a corresponding business data table according to the correspondence between the business file identifier and the business data. The file cleaning component is connected with the database and the MongoDB file server.

[0114] As shown in Figure 8 , the method for cleaning up invalid files specifically includes the following steps:

[0115] First, the file cleaning component acquires the modification time of each business file stored in the MongoDB in response to a file cleaning instruction, determines the modification interval duration of each business file according to the modification time and the current time, determines the business file with a modification interval duration exceeding a preset duration threshold as a to-be-cleaned file, and sets the usage identifier inUse of each to-be-cleaned file to N.

[0116] Subsequently, the Mapper or Dao that implements the file selection interface is obtained through the Spring context ApplicationContext, and the corresponding processing thread is assigned to each Mapper or Dao. Meanwhile, the streaming query ResultHandler of Mybatis is used to call the processing thread corresponding to each Mapper or Dao in batches according to the preset call quantity, the file selection interface corresponding to each Mapper or Dao is called through the processing thread, and the valid file ID associated with the business data in each business data table corresponding to the thread is obtained. Specifically, the business to which each business data table belongs implements the selectFileIds method of the CommonFileDao interface according to its own valid file selection logic. The interface method is used to obtain the valid fileId in the related business data table. After all batches of processing threads complete the task of obtaining the valid file ID, the valid file IDs obtained in each batch are collected to obtain the valid file IDs associated with each business data.

[0117] Each processing thread continues to match each valid file ID with the file ID of the file to be cleaned in MongoDB. If the matching is successful, the inUse of the file to be cleaned is modified from N to Y. After all processing threads complete the matching task, the file to be cleaned in MongoDB whose inUse is still N is determined as an invalid file.

[0118] The invalid file cleaning instruction is generated according to the file ID of the invalid file, the invalid file cleaning instruction is sent to MongoDB, MongoDB is controlled to delete the GridFS file corresponding to the invalid file, and each invalid file is deleted.

[0119] The invalid file cleaning method in the embodiment can accurately clean the invalid files, solves the invalid file identification problem by adding the file use identifier inUse to the file, separates the abstraction and implementation of the valid function by using the bridge mode, realizes the extensible multiple business data files, and realizes the validity of each business data file. The problem that the judgment logic of each business data valid file is different and the field of the associated file id is different is solved. By using the multi-thread parallel combination of ResultHandler and MongoDB batch update method, high concurrency is realized, and the component running efficiency is improved.

[0120] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0121] Based on the same inventive concept, the embodiments of the present application also provide a failed file cleaning device for implementing the above-mentioned failed file cleaning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more failed file cleaning device embodiments provided below can refer to the limitations of the failed file cleaning method described above, which will not be repeated here.

[0122] In one embodiment, as shown in Figure 9 A failed file cleaning device 900 is provided, comprising: an instruction response module 901, an effective file determination module 902, a failed file determination module 903, and a cleaning module 904, wherein:

[0123] The instruction response module 901 is configured to determine the files to be cleaned in the file server in response to the file cleaning instruction.

[0124] The effective file determination module 902 is configured to obtain the business file identifiers associated with each business data through the file selection interface of each business data table, and determine the business file identifiers associated with each business data as effective file identifiers; the business data table is configured to store the correspondence between each business data and each business file.

[0125] The failed file determination module 903 is configured to match the files to be cleaned according to the effective file identifiers, and determine the files to be cleaned that fail to match the effective file identifiers as failed files.

[0126] The cleaning module 904 is configured to generate a failed file cleaning instruction based on the failed files, and control the file server to clean the failed files.

[0127] The invalid file cleaning device determines the to-be-cleaned files in the file server in response to a file cleaning instruction, and obtains the business file identifiers associated with the business data from the business data tables through the file selection interfaces of the business data tables, according to the correspondence between the business data and the business files stored in the business data tables. If the business files are associated with the business data, the business files are determined as valid files, and the business file identifiers associated with the business data are determined as valid file identifiers. The to-be-cleaned files are matched with the valid file identifiers, and the to-be-cleaned files that fail to match the valid file identifiers are determined as invalid files. The to-be-cleaned files are screened again by using the valid file identifiers, so that the accuracy of the invalid file determination is greatly improved. Then, the invalid file cleaning instruction is generated according to the accurate invalid files, and the file server is controlled to clean the invalid files, so that the storage space of the file server is effectively saved.

[0128] In one embodiment, the instruction response module is further configured to: obtain modification times of the business files stored in the file server; determine modification interval durations of the business files according to the modification times and a current time; and determine the business files with the modification interval durations exceeding a preset duration threshold as the to-be-cleaned files.

[0129] In one embodiment, the valid file determination module is further configured to: call a file selection interface pre-set in a data access layer of the business data table to obtain the business file identifiers associated with the business data; and implement the file selection interface by the business to which the data access layer belongs, to perform valid file selection on the business data table.

[0130] In one embodiment, the valid file determination module is further configured to: obtain data access layers of the business data tables, the data access layers being pre-set with the file selection interface; allocate processing threads to the data access layers according to the business to which the data access layers belong; and call the file selection interface of the data access layers based on the processing threads, to instruct the business to which the data access layers belong to implement the file selection interface, so as to perform valid file selection on the business data table and obtain the business file identifiers associated with the business data.

[0131] In one embodiment, the invalid file determination module is further configured to: match the valid file identifiers obtained by the processing threads with the file identifiers of the to-be-cleaned files through the processing threads; set valid use identifiers for the to-be-cleaned files that are matched successfully in the processing threads; and determine the to-be-cleaned files without the valid use identifiers as the invalid files after the processing threads are matched.

[0132] In an embodiment, the effective file determination module is further configured to: use a stream query database framework to call the file selection interface preset in the data access layer of each business data table in batches according to a preset calling quantity; receive a batch business file identifier fed back by the file selection interface corresponding to the business implementation to which each batch data access layer belongs; and collect each batch business file identifier to obtain a business file identifier associated with each business data.

[0133] The modules in the invalid file cleaning device described above can be implemented wholly or partially by software, hardware, and combinations thereof. The modules described above can be embedded in a processor in a computer device in hardware form or independent of the processor in the computer device, or can be stored in a memory in the computer device in software form to be called and executed by the processor to perform operations corresponding to the modules.

[0134] In an embodiment, a computer device is provided, which can be a server integrated with a file cleaning component, and an internal structure diagram of the computer device can be as shown in Figure 10 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data such as a file selection interface and a business file identifier. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an invalid file cleaning method.

[0135] Those skilled in the art can understand that Figure 10 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In an embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps:

[0137] In response to a file cleaning instruction, determining a file to be cleaned in a file server;

[0138] The file selection interface of each business data table is called to obtain the business file identifiers associated with the business data, and the business file identifiers associated with the business data are determined as valid file identifiers; the business data table is used to store the correspondence between each business data and each business file;

[0139] The to-be-cleaned files are matched according to the valid file identifiers, and the to-be-cleaned files that fail to match the valid file identifiers are determined as invalid files;

[0140] The invalid file cleaning instruction is generated based on the invalid files, and the file server is controlled to clean the invalid files.

[0141] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0142] The modification time of each business file stored in the file server is obtained;

[0143] The modification interval duration of each business file is determined according to the modification time and the current time;

[0144] The business file whose modification interval duration exceeds the preset duration threshold is determined as a to-be-cleaned file.

[0145] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0146] The file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifiers associated with the business data; the business file identifiers are obtained by each data access layer belonging to a business respectively implementing a corresponding file selection interface, and effective file selection is performed on the business data table.

[0147] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0148] The data access layer of each business data table is obtained, and the data access layer is pre-provided with the file selection interface;

[0149] According to the business to which each data access layer belongs, a processing thread is allocated to each data access layer;

[0150] Based on the file selection interface of each data access layer called by each processing thread, each data access layer belonging to a business respectively implements the file selection interface, and effective file selection is performed on the business data table to obtain the business file identifiers associated with each business data.

[0151] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0152] Each processing thread respectively matches the valid file identifiers obtained by each processing thread with the file identifiers of each to-be-cleaned file;

[0153] setting a valid use identifier for the to-be-cleaned file matched successfully in each processing thread;

[0154] After the matching of each processing thread is completed, the to-be-cleaned file without the valid use identifier is determined as an invalid file.

[0155] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0156] using a stream query database framework, calling the file selection interface preset in the data access layer of each business data table in batches according to a preset calling quantity;

[0157] receiving the batch business file identifier fed back by the file selection interface corresponding to the business implementation to which each batch data access layer belongs;

[0158] collecting each batch business file identifier to obtain the business file identifier associated with each business data.

[0159] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program, when executed by the processor, implements the following steps:

[0160] in response to a file cleaning instruction, determining a to-be-cleaned file in a file server;

[0161] obtaining, through the file selection interface of each business data table, a business file identifier associated with each business data, and determining the business file identifier associated with each business data as a valid file identifier; the business data table is used to store the correspondence between each business data and each business file;

[0162] matching the to-be-cleaned file according to the valid file identifier, and determining the to-be-cleaned file that fails to match the valid file identifier as an invalid file;

[0163] generating an invalid file cleaning instruction based on the invalid file, and controlling the file server to clean the invalid file.

[0164] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0165] obtaining the modification time of each business file stored in the file server;

[0166] determining the modification interval duration of each business file according to the modification time and the current time;

[0167] determining the business file with a modification interval duration exceeding a preset duration threshold as a to-be-cleaned file.

[0168] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0169] The file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifiers associated with each business data; the business file identifiers are obtained by each business corresponding to the file selection interface implemented by the business to which each data access layer belongs, and effective file selection is performed on the business data table.

[0170] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0171] Obtain the data access layer of each business data table, and the data access layer is pre-set with a file selection interface;

[0172] According to the business to which each data access layer belongs, a processing thread is allocated to each data access layer;

[0173] Based on each processing thread, the file selection interface of each data access layer is called to instruct the business to which each data access layer belongs to implement the file selection interface respectively, and effective file selection is performed on the business data table to obtain the business file identifiers associated with each business data.

[0174] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0175] Through each processing thread, the effective file identifiers obtained by each processing thread are matched with the file identifiers of each file to be cleaned respectively;

[0176] An effective use identifier is set for the file to be cleaned that is matched successfully in each processing thread;

[0177] After each processing thread is matched, the file to be cleaned that does not have an effective use identifier is determined as an invalid file.

[0178] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0179] Using a stream query database framework, the file selection interface pre-set in the data access layer of each business data table is called in batches according to a preset calling quantity;

[0180] After receiving the batch business file identifiers fed back by the business corresponding to the file selection interface implemented by each batch data access layer;

[0181] The batch business file identifiers are collected to obtain the business file identifiers associated with each business data.

[0182] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0183] In response to a file cleaning instruction, determine the file to be cleaned in the file server;

[0184] The file selection interface of each business data table is called to obtain the business file identifiers associated with the business data, and the business file identifiers associated with the business data are determined as valid file identifiers; the business data table is used to store the correspondence between each business data and each business file;

[0185] The to-be-cleaned files that fail to match the valid file identifiers are determined as invalid files;

[0186] The invalid file cleaning instruction is generated based on the invalid files, and the file server is controlled to clean the invalid files.

[0187] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0188] The modification time of each business file stored in the file server is obtained;

[0189] The modification interval duration of each business file is determined according to the modification time and the current time;

[0190] The business file whose modification interval duration exceeds the preset duration threshold is determined as the to-be-cleaned file.

[0191] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0192] The file selection interface pre-set in the data access layer of each business data table is called to obtain the business file identifiers associated with the business data; the business file identifiers are obtained by the business to which each data access layer belongs respectively implementing the corresponding file selection interface, and the valid file selection of the business data table is performed.

[0193] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0194] The data access layer of each business data table is obtained, and the data access layer is pre-provided with the file selection interface;

[0195] The processing thread is allocated to each data access layer according to the business to which each data access layer belongs;

[0196] The file selection interface of each data access layer is called based on each processing thread, the business to which each data access layer belongs is instructed to respectively implement the file selection interface, the valid file selection of the business data table is performed, and the business file identifiers associated with each business data are obtained.

[0197] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0198] The valid file identifier obtained by each processing thread is matched with the file identifier of each file to be cleaned respectively by each processing thread;

[0199] An effective use identifier is set for the file to be cleaned which is matched successfully in each processing thread;

[0200] After the matching of each processing thread is completed, the file to be cleaned without the effective use identifier is determined as an invalid file.

[0201] In one embodiment, the computer program is further implemented to perform the following steps when executed by the processor:

[0202] The file selection interface of the data access layer pre-set in each business data table is called in batches according to a preset calling quantity using a stream query database framework;

[0203] The batch business file identifier fed back by the file selection interface corresponding to the business implementation to which each batch data access layer belongs is received;

[0204] The batch business file identifiers are collected to obtain the business file identifiers associated with each business data.

[0205] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0206] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0207] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0208] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for purging obsolete files, characterized by, The method comprises: in response to a file cleaning instruction, determining a to-be-cleaned file in a file server; calling a file selection interface pre-set in a data access layer of each business data table to obtain a business file identifier associated with each business data; the business file identifier is determined by a corresponding file selection interface implemented by a business to which each data access layer belongs, and effective file selection is performed on the business data table to obtain a valid file identifier; the business data table is used to store a corresponding relationship between each business data and each business file; matching the to-be-cleaned file according to the valid file identifier, and determining a to-be-cleaned file that fails to match the valid file identifier as an invalid file; generating an invalid file cleaning instruction based on the invalid file, and controlling the file server to clean the invalid file.

2. The method of claim 1, wherein, The method comprises: obtaining a modification time of each business file stored in the file server; determining a modification interval duration of each business file according to the modification time and a current time; determining a business file whose modification interval duration exceeds a preset duration threshold as a to-be-cleaned file.

3. The method of claim 1, wherein, The method comprises: obtaining a data access layer of each business data table, the data access layer being pre-provided with a file selection interface; allocating a processing thread to each data access layer according to a business to which each data access layer belongs; calling a file selection interface of each data access layer based on each processing thread, instructing a business to which each data access layer belongs to implement the file selection interface respectively, performing effective file selection on the business data table, and obtaining a business file identifier associated with each business data.

4. The method of claim 3, wherein, The method comprises: matching, through each processing thread, the valid file identifier obtained by each processing thread with a file identifier of each to-be-cleaned file; setting a valid use identifier for a to-be-cleaned file that matches successfully in each processing thread; after each processing thread is matched, determining a to-be-cleaned file that does not have the valid use identifier as an invalid file.

5. The method according to any one of claims 1 to 4, characterized in that, The method comprises: using a stream query database framework to call the file selection interface pre-provided in the data access layer of each business data table in batches according to a preset calling quantity; receiving batch business file identifiers fed back after a business to which each batch data access layer belongs implements a corresponding file selection interface; collecting each batch business file identifier to obtain a business file identifier associated with each business data.

6. An invalid file cleaning apparatus characterized by comprising: The device comprises: an instruction response module configured to determine a to-be-cleaned file in a file server in response to a file cleaning instruction; An effective file determination module is configured to call a file selection interface preset in a data access layer of each service data table, and obtain service file identifiers associated with each service data; the service file identifiers are determined by corresponding file selection interfaces of services to which each data access layer belongs, and the service data table is subjected to effective file selection to obtain the service file identifiers associated with each service data as effective file identifiers; the service data table is configured to store a corresponding relationship between each service data and each service file; An invalid file determination module is configured to match the effective file identifiers with the files to be cleaned, and determine files to be cleaned that fail to match the effective file identifiers as invalid files; A cleaning module is configured to generate an invalid file cleaning instruction based on the invalid files, and control the file server to clean the invalid files.

7. The apparatus of claim 6, wherein, The instruction response module is further configured to: obtain modification times of each service file stored in the file server; determine modification interval durations of each service file according to the modification times and a current time; determine service files with modification interval durations exceeding a preset duration threshold as files to be cleaned.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images