Operator-based File Type Screening Method, Device, and Computer Storage Medium

Abstract

The present application discloses an operator-based file type screening method, device, and computer storage medium, relating to the technical field of text data processing. The method includes: generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder; obtaining the operator adaptation type of the current operator, classifying the set of files to be processed according to whether the file type matches the operator adaptation type, classifying the adapted files to the source folder, and classifying the unadapted files to the subsequent source folder; when executing the current operator, obtaining the files that need to be executed from the source folder, and storing the processed files in the target folder; after the execution of the current operator is completed, continuing to execute the next operator until the processing flow of all operators is completed. The present application creates corresponding folders for operators and classifies files according to the operator adaptation type, accurately matching the processing requirements and avoiding resource waste.

Description

Technical Field

[0001] This application relates to the technical field of text data processing, and in particular, to a method, device, and computer storage medium for screening file types based on operators. Background Art

[0002] With the development of linguistics-related research and applications such as Natural Language Processing (NLP), text mining, and information retrieval, corpus processing has become increasingly important. Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on a large-scale text data set (i.e., a corpus), aiming to extract useful information from a large amount of text data to support numerous applications. Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction, and final analysis, and involves various file types, such as plain text files, rich text format files, web documents, emails, social media posts, etc., and even non-traditional text data sources such as images and videos.

[0003] In corpus processing, an "operator" refers to a reusable processing unit or function that undertakes different tasks such as data reading, conversion, cleaning, and feature extraction. There are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc. However, when actually processing a corpus containing multiple types of files, the files are often directly sent to the operators for execution without being screened and classified. It should be clear that an operator cannot adapt to all types of files. For example, an image operator cannot process text files. Therefore, when an operator encounters an unmatched file type, it is likely to cause the operator to perform ineffective processing on the unmatched file type, which not only fails to generate useful result files but also causes waste of resources. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, and computer storage medium for screening file types based on operators, aiming to solve the technical problem that in corpus processing, the files are not screened and classified, resulting in ineffective processing of unmatched file types by the operators and the inability to generate useful result files.

[0005] To achieve the above objective, an embodiment of this application provides a method for screening file types based on operators, and the method for screening file types based on operators includes:

[0006] Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, a subsequent source folder, and a target folder under the folder;

[0007] Obtain the operator adaptation type of the current operator, classify the set of files to be processed according to whether the file type matches the operator adaptation type, classify the adapted files into the source folder, and classify the unadapted files into the subsequent source folder;

[0008] When executing the current operator, obtain the files to be executed from the source folder and store the processed files in the target folder;

[0009] After the current operator is executed, continue to execute the next operator until the processing flow of all operators is completed.

[0010] In an embodiment, before the step of obtaining the operator adaptation type of the current operator, classifying the set of files to be processed according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the unadapted files into the subsequent source folder, it includes:

[0011] Obtain the file name of each file in the set of files to be processed;

[0012] Extract the file suffix of each file name and determine the file type of each file based on the file suffix.

[0013] In an embodiment, before the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder, and a target folder under the folder, it includes:

[0014] Receive an operator creation instruction, where the operator creation instruction includes an operator instance identifier and the file type adapted by the operator;

[0015] According to the operator creation instruction, store the correspondence between the operator and the file type.

[0016] In an embodiment, the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder, and a target folder under the folder further includes:

[0017] According to the processing requirements, determine the operator sequence required for the set of files to be processed;

[0018] According to the operator sequence, generate a corresponding folder for each operator in turn, and generate a source folder, a subsequent source folder, and a target folder under each folder.

[0019] In an embodiment, the step of, when executing the current operator, obtaining the files to be executed from the source folder and storing the processed files in the target folder includes:

[0020] When executing the current operator, obtain the files to be executed from the source folder corresponding to the current operator;

[0021] Execute the preset processing operation corresponding to the current operator on each file to be executed;

[0022] Store each file after the preset processing operation in the target folder corresponding to the current operator.

[0023] In one embodiment, the steps of obtaining the files to be executed from the source folder and storing the processed files in the target folder when executing the current operator further include:

[0024] If the source folder corresponding to the current operator is empty, skip the execution of the current operator and enter the execution process of the next operator.

[0025] In one embodiment, the step of continuing to execute the next operator after the current operator is executed until all operator processing processes are completed includes:

[0026] After the current operator is executed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the step of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder.

[0027] In one embodiment, after the step of re-executing the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder, and a target folder under the folder when the next operator is executed, it further includes:

[0028] Based on the subsequent source folder and the target folder corresponding to the previous operator, re-execute the step of classifying according to whether the file type matches the operator adaptation type;

[0029] Continue to execute the next operator based on the re-classified source folder until all operator processing processes are completed.

[0030] An embodiment of the present application further provides a file type screening device based on an operator. The file type screening device based on an operator includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the file type screening method based on an operator as described above.

[0031] The embodiment of the present application also provides a computer storage medium. The computer storage medium is a computer-readable storage medium, and a computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the file type screening method based on an operator as described above are implemented.

[0032] The embodiment of the present application discloses a file type screening method based on an operator. By generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder; obtaining the operator adaptation type of the current operator, classifying the set of files to be processed according to whether the file type matches the operator adaptation type, classifying the adapted files to the source folder, and classifying the unadapted files to the subsequent source folder; when executing the current operator, obtaining the files to be executed from the source folder, and storing the processed files in the target folder; after the current operator is executed, continuing to execute the next operator until all operator processing processes are completed. The present application creates corresponding folders for operators and classifies files according to the operator adaptation type, accurately matching the processing requirements and avoiding resource waste. Brief Description of the Drawings

[0033] Figure 1 is a flowchart of the first embodiment of the file type screening method based on an operator involved in the solution of the embodiment of the present application;

[0034] Figure 2 is a flowchart of the second embodiment of the file type screening method based on an operator involved in the solution of the embodiment of the present application;

[0035] Figure 3 is a flowchart of the third embodiment of the file type screening method based on an operator involved in the solution of the embodiment of the present application;

[0036] Figure 4 is a flowchart of the fourth embodiment of the file type screening method based on an operator involved in the solution of the embodiment of the present application;

[0037] Figure 5 is a structural diagram of the file type screening device based on an operator involved in the solution of the embodiment of the present application.

[0038] The implementation, functional features, and advantages of the object of the present application will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0039] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] With the development of linguistics-related research and applications such as Natural Language Processing (NLP), text mining, and information retrieval, corpus processing has become increasingly important. Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on large-scale text datasets (i.e., corpora), aiming to extract useful information from a large amount of text data to support numerous applications. Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction, and final analysis, and involves various file types, such as plain text files, rich text format files, web documents, emails, social media posts, etc., and even includes non-traditional text data sources such as images and videos.

[0041] In corpus processing, an "operator" refers to a reusable processing unit or function that undertakes different tasks such as data reading, transformation, cleaning, feature extraction, etc. There are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc. However, when actually processing a corpus containing multiple types of files, the files are often directly sent to the operator for execution without being screened and classified. It should be clear that an operator cannot adapt to all types of files. For example, an image operator cannot process text files. Therefore, when an operator encounters a file type that does not match, it is easy to cause the operator to perform ineffective processing on the inadaptable file type, which not only fails to generate useful result files but also causes waste of resources.

[0042] To solve the above-mentioned defects existing in the related technologies, an embodiment of the present application proposes a file type screening method based on an operator. This method generates a folder according to the operator instance identifier of the current operator, and generates a source folder, a subsequent source folder, and a target folder under the folder; obtains the operator adaptation type of the current operator, classifies the set of files to be processed according to whether the file type matches the operator adaptation type, and classifies the adaptable files into the source folder and the inadaptable files into the subsequent source folder; when executing the current operator, obtains the files that need to be executed from the source folder and stores the processed files in the target folder; after the current operator is executed, continues to execute the next operator until all operator processing processes are completed. The present application creates corresponding folders for operators and classifies files according to the operator adaptation type, accurately matching the processing requirements and avoiding waste of resources.

[0043] It should be noted that the execution subject of this embodiment can be an operator-based file type screening system, or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an operator-based file type screening device that can implement the above functions. Hereinafter, an operator-based file type screening system (hereinafter referred to as "system") will be taken as an example to illustrate this embodiment and the following embodiments.

[0044] For the operator-based file type screening method according to the first embodiment proposed in this application, please refer to Figure 1 , and this method includes steps S10 to S40:

[0045] Step S10: Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, a subsequent source folder, and a target folder under the folder.

[0046] Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on a large-scale text dataset (i.e., a corpus), aiming to extract useful information from a large amount of text data to support numerous applications such as natural language processing (NLP), text mining, information retrieval, and other linguistic research.

[0047] Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction, and final analysis, and each stage requires corresponding technical means and tools.

[0048] In actual operation, corpus processing involves various file types, such as plain text files (such as ".txt"), rich text format files (such as ".docx", ".pdf"), web documents, emails, social media posts, etc., and even includes non-traditional text data sources such as images and videos. During the processing, different file types require different operators to complete specific tasks, such as text cleaning, word segmentation, part-of-speech tagging, named entity recognition, etc.

[0049] In addition, with the increasing amount of multimedia content, corpus processing also includes the processing of non-traditional text data sources such as processing images through optical character recognition (OCR) and processing videos through video frame conversion and in-frame text recognition.

[0050] It should be noted that in corpus processing, an "operator" refers to a reusable processing unit or function that undertakes different tasks such as data reading, data conversion, data cleaning, feature extraction, analysis and processing, and result output. An operator can be regarded as a node or module in the data processing process, and each node has a specific processing task.

[0051] When processing a multimedia data set, the concept of an "operator" can be extended to specific processing steps or functional modules for different types of files, that is, there are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc.

[0052] Exemplarily, in the text file processing operator, the function of the text cleaning operator is to remove non-text content (such as HTML tags) in the text and normalize the text format. The input is a text file, and the output is clean text content. In the PDF file processing operator, the function of the PDF text extraction operator is to extract text content from a PDF file. The input is a PDF file, and the output is the text content in the PDF file. In the image file processing operator, the function of the OCR operator is to convert the text in the image into text format. The input is an image file, and the output is the text content extracted from the image file. In the video file processing operator, the function of the video frame conversion operator is to convert a video file into a series of image frames. The input is a video file, and the output is a sequence of image frames.

[0053] Different operators have different requirements for file types. For example, an image file processing operator cannot process a text file. If the file set to be processed is not screened and classified, but directly sent to the operator for execution, no useful result file can be generated between the incompatible file and the operator, but the operator will still execute, resulting in waste of resources and prone to error reporting, leading to unstable services.

[0054] In this embodiment, when each operator executes, an operator instance will be generated, and the operator instance identifier can be generated using the Snowflake algorithm. The Snowflake algorithm is an algorithm for generating unique identifiers, which can ensure that each operator instance has a unique identifier in the entire system, avoiding confusion and conflicts during the processing process.

[0055] When the operator starts to execute, the system will create a folder based on the operator instance identifier, which is the basic storage unit for each subsequent operator when processing the file set to be processed. Under this folder, three sub-folders will be further generated, namely the source folder (source), the subsequent source folder (nextSource), and the target folder (target).

[0056] The source folder is used to store the files that the current operator can process. It is the starting point for the operator to begin processing files and contains the files to be processed that are adapted to the current operator. The subsequent source folder is used to store those files that are outside the processing scope of the current operator (i.e., the operator adaptation types that do not match the current operator), but may be used later in the entire corpus processing process, providing a staging or backup space for processing operations at different stages. The target folder is used to store the processing results of the current operator. After the current operator completes the processing operation on the files, the final processing results will be stored in the target folder for subsequent viewing, analysis, or further use.

[0057] The above method of generating corresponding folders and subfolders for each operator ensures that in a complex corpus processing process, different types of files can be processed and stored orderly under the processing of each operator. This makes the file processing processes between different operators independent of each other, avoiding file confusion between different operator operations. In addition, it also helps in the reasonable allocation and management of file resources. By classifying files according to the processing stages, it is convenient to search for and use files during the processing, improving the stability and operability of the entire corpus processing system.

[0058] Step S20: Obtain the operator adaptation type of the current operator, classify the set of files to be processed according to whether the file type matches the operator adaptation type, classify the adapted files into the source folder, and classify the non-adapted files into the subsequent source folder.

[0059] In this embodiment, the set of files to be processed is a group of files that need to be processed for the corpus, which may contain various different file types. The operator adaptation type defines the range of file types that the current operator can process. When the user creates an operator, the file types that the operator is adapted to will be determined in advance. Therefore, when the user creates an operator, the operator adaptation type of the operator will be identified by specifying the label of the operator. According to the label of the operator, the operator adaptation type of the current operator can be obtained.

[0060] Exemplarily, for the text data cleaning operator, it is suitable for file types such as "jsonl", "json", "txt", "csv", "xlsx", "xls", etc. Then the name of this operator can be expressed as text data cleaning, the type of the label is CLEANTXT, and the operator adaptation type of this operator includes "jsonl", "json", "txt", "csv", "xlsx", "xls". When the current operator is text data cleaning, if a certain file in the set of files to be processed is of the picture type, then according to the operator adaptation type of the current operator, it can be known that the file of the picture type is not suitable for the current operator, and it will be placed in the subsequent source folder under this operator.

[0061] In this embodiment, the three sub-folders, namely the source folder, the subsequent source folder, and the target folder, are used to store the files after classifying the file set to be processed according to file types and the result files after being processed by the operator.

[0062] In an alternative implementation, before step S20, steps S201 to S202 are included:

[0063] Step S201: Obtain the file name of each file in the file set to be processed.

[0064] Step S202: Extract the file suffix of each said file name, and determine the file type of each file based on the file suffix.

[0065] In this embodiment, the file type is obtained according to the file extension or file header information.

[0066] Specifically, use a file operation library to read a part of the content of each file in the file set to be processed, and judge the file type according to its format characteristics. For common file types (such as ".txt", ".jpg", ".pdf", etc.), the file type can also be judged by the file extension. By extracting the file suffix in the file name, the file type of the file is determined.

[0067] Then, each file in the file set to be processed is compared with the operator adaptation type of the currently executing operator to judge whether it matches, and each file in the file set to be processed is classified.

[0068] When classifying each file in the file set to be processed, the file operation function can be used to move or copy the file to the corresponding folder. Taking Python as an example, the copy or move function in the shutil module can be used. For the files that match the operator adaptation type of the current operator, the files are copied or moved to the source folder; for the files that do not match, the files are copied or moved to the subsequent source folder.

[0069] Through the above steps, the files in the file set to be processed will be stored in different sub-folders according to the matching situation of the file types, providing a clear file classification and storage structure for the subsequent operator execution and processing, which helps to improve the manageability and efficiency of corpus processing.

[0070] Step S30: When executing the current operator, obtain the files that need to be executed from the source folder, and store the processed files in the target folder.

[0071] In this embodiment, when the set of files to be processed is classified, the current operator is executed. In the source folder corresponding to the current operator, the files stored are those classified as matching the operator adaptation type of the current operator in the previous steps, and these files are the objects to be processed by the current operator.

[0072] Specifically, first traverse the files in the source folder. For each file, open and read the file content in read mode, and then use the current operator to process these files. The specific processing operations depend on the specific functions and implementation methods of the current operator. Finally, store the processed results in the target folder.

[0073] Exemplarily, if the current operator is a data cleaning operator, the data in the file will be cleaned. If the current operator is a format conversion operator, the format of the file will be converted. After the current operator finishes processing the file, the processed file will be stored in the target folder. Among them, the file operation library can be used to store the processed file. For example, in Python, use the open function to open the file in the target folder in write mode, and then use the write method to write the content of the processed file into this file.

[0074] It should be noted that using the open function to open the file in write mode is to create or open a file and write the processed content into it.

[0075] Step S40: After the current operator finishes execution, continue to execute the next operator until the processing flow of all operators is completed.

[0076] In this embodiment, the set of files to be processed usually includes files of multiple different file types. Therefore, during the corpus processing, due to the diversity of file types, multiple operators are usually required to process the set of files to be processed. Each operator undertakes a specific function and jointly completes a complex file processing task. These operators will be executed in sequence according to a preset order to ensure the orderliness and coherence of the entire file processing flow.

[0077] The current operator refers to the operator that is currently performing an operation. After the current operator finishes execution, the system will continue to execute the next operator until all operators are executed, thus completing the entire corpus processing flow of the set of files to be processed.

[0078] In an optional implementation, create a queue or a list for setting all the operators that need to be executed in the corpus processing flow. Then, use a scheduler or other scheduling mechanisms to execute the corresponding operators in sequence according to the order of the queue or the list. During the execution of the operators, track the execution status of each operator to determine whether there are other operators that need to be executed after the current operator finishes execution.

[0079] Please refer to Figure 2 , for the operator-based file type screening method of the second embodiment proposed in this application, before step S10, it includes steps S101 to S102:

[0080] Step S101: Receive an operator creation instruction, where the operator creation instruction includes an operator instance identifier and a file type adapted by the operator.

[0081] Step S102: According to the operator creation instruction, store the correspondence between the operator and the file type.

[0082] When performing corpus processing, it is necessary to ensure that each file can be correctly processed. Therefore, the user will configure corresponding operators for each file type. During the process of the user creating an operator, the operator adaptation type of the corresponding operator is identified by specifying the label of the operator.

[0083] After the system receives the operator creation instruction, it is necessary to store the correspondence between the operator and the file type adapted by the operator. When the operator is executed subsequently, the system can determine the file range processed by the current operator through the mapping relationship, avoiding incorrect matching of operators and file types during the processing, and ensuring the accuracy and efficiency of file processing. In addition, when it is necessary to add or modify an operator, only the stored correspondence needs to be updated to integrate the new operator into the entire corpus processing process, providing a necessary information basis for the subsequent orderly screening and processing of files.

[0084] Please refer to Figure 3 , for the operator-based file type screening method of the third embodiment proposed in this application, step S10 further includes steps S110 to S120:

[0085] Step S110: According to the processing requirements, determine the operator sequence required to be executed by the set of files to be processed.

[0086] In this embodiment, the processing requirements are only the specific processing objectives of the set of files to be processed. When processing the set of files to be processed, different file types and different processing requirements mean that different operators need to be used. For example, to extract text information from a picture, it may be necessary to first go through an OCR operator, while for a video, it may be necessary to first convert it into a series of frame images and then process the text content in each frame separately.

[0087] Therefore, reasonable operator scheduling can ensure that each file is properly processed during the processing, so as to achieve the expected processing effect. Operator scheduling is the orderly arrangement and scheduling of the operators required for a file. For a set of files to be processed, the user clearly knows the processing effect to be achieved. Then, which operator to execute first and which operator to execute later can be determined according to the processing requirements.

[0088] These operators are arranged in the order of processing requirements to obtain an operator sequence.

[0089] Step S120: According to the operator sequence, generate corresponding folders for each operator in turn, and generate a source folder, a subsequent source folder, and a target folder under each folder.

[0090] Since different operators need independent spaces to store various files during file processing to ensure the orderliness and independence of operations. Therefore, in this embodiment, corresponding folders are generated for each operator, making the folder the basic storage unit for the operator to process files. Under this basic storage unit, a source folder, a subsequent source folder, and a target folder will be further generated.

[0091] In this embodiment, in order to meet diverse processing requirements in complex file processing tasks, the processing process needs to be decomposed into multiple tasks and executed sequentially by different operators. The operation of each operator requires an independent storage area to avoid file confusion and operation interference between different operators.

[0092] By the above method, allocating independent folders and subfolders for each operator helps to clearly organize files, facilitates the management and monitoring of the processing process of each operator, and improves the efficiency and accuracy of file processing. At the same time, the design of the above storage structure also facilitates the distinction and preservation of the processing results of different operators, helps to ensure the independence and integrity of files in different processing stages, provides a good foundation for subsequent file screening, processing, and storage operations, and meets the file processing tasks under different processing requirements.

[0093] Please refer to Figure 4 , the method for screening file types based on operators according to the fourth embodiment proposed in this application, step S30 includes steps S310 to S330:

[0094] Step S310: When executing the current operator, obtain the files to be executed from the source folder corresponding to the current operator.

[0095] In this embodiment, when executing the current operator, the files to be executed are obtained from the source folder corresponding to the current operator. Here, the source folder refers to the folder created for the current operator when the current operator is executed, and it stores the files adapted to the operator adaptation type of the current operator. These files are the files that meet the processing scope of the current operator after being classified in the foregoing steps, and provide the input data source for the operation of the current operator.

[0096] Step S320: Perform the preset processing operation corresponding to the current operator on each file to be executed.

[0097] In this embodiment, the preset processing operation is an operation set in advance according to the function and task of the current operator. For example, for text files, the preset processing operation may be data cleaning, format conversion, or information extraction. For image files, it may be operations such as image enhancement, size adjustment, or format conversion. Each operator has its specific preset processing operation, aiming to perform specific processing on the files to meet different processing requirements.

[0098] Step S330: Store each file after completing the preset processing operation in the target folder corresponding to the current operator.

[0099] The target folder stores the result files processed by the current operator. Storing the processed files in the target folder is, on the one hand, to centrally manage the processing results for convenient subsequent viewing and use; on the other hand, it also helps to maintain the orderliness of the entire file processing process and avoid confusion between the processing results and other operators or operations.

[0100] In the above manner, each operator has a corresponding file processing path, obtains the input file from the source folder, undergoes its own processing operation, and finally stores the result in the target folder.

[0101] Further, step S30 further includes step S340:

[0102] Step S340: If the source folder corresponding to the current operator is empty, skip the execution of the current operator and enter the execution process of the next operator.

[0103] In this embodiment, if an operator without files to process is forced to perform an operation, it will not only waste system resources but may also cause errors or abnormal situations. Therefore, if it is found that the source folder corresponding to the current operator is empty, the execution of the current operator is skipped and the execution process directly enters the next operator. In this way, the entire file processing process can be made more efficient and smooth, ensuring that the operator only operates on the cases where there are actual files to be processed, and for operators without files to process, the processing process is directly transferred to the next operator to ensure the coherence and effectiveness of the entire file processing process.

[0104] The operator-based file type screening method according to the fifth embodiment proposed in this application, step S40 includes step S410:

[0105] Step S410: After the execution of the current operator is completed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the step of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder.

[0106] When executing the corpus processing or file processing flow, an operator sequence will be preset in advance to determine the order of execution of each operator during the processing. If the current operator has been executed (including when the source folder corresponding to the current operator is empty and the execution operation of the current operator is skipped), it is necessary to check whether there are subsequent operators that need to continue processing the file. If there is a next operator, then continue with the execution of the next operator. When the next operator is executed, the operations of generating a folder according to the operator instance identifier and generating corresponding sub-folders (source folder, subsequent source folder, and target folder) will be re-executed. This is to provide an independent file storage and operation environment for the next operator to ensure the orderliness and independence of the processing process of the next operator.

[0107] Further, after step S410, steps S420 to S430 are further included:

[0108] Step S420: Based on the subsequent source folder and the target folder corresponding to the previous operator, re-execute the step of classifying according to whether the file type matches the operator adaptation type.

[0109] In this embodiment, in the subsequent source folder of the previous operator, files that failed to be adapted to the previously executed operators in the previous operations are stored, and the target folder of the previous operator stores the result files after the previous operator's processing.

[0110] After the execution of the current operator is completed, continue with the execution of the next operator according to the operator sequence. When the next operator is executed, it is necessary to re-perform the classification operation of the file type and the operator adaptation type, and screen the files in the subsequent source folder and the target folder of the previous operator again according to the adaptation type of the new operator (i.e., the next operator) to prepare a suitable file source for the operation of the next operator. Then save the classification result to the sub-folder corresponding to the new operator again.

[0111] This embodiment ensures the coherence and adaptability of file processing between different operators, enabling the files that the previous operator failed to process and the files after processing to be reasonably processed and allocated according to the adaptation situation of the new operator.

[0112] Step S430: Based on the reclassified source folder, continue to execute the next operator until all operator processing flows are completed.

[0113] This embodiment is a promotion mechanism for the file processing flow, ensuring that the entire file processing process can proceed orderly according to the operator sequence. After each operator processes the file, it can pass the processed file and the unprocessed file to the next operator, and reasonably redistribute and process the file resources until the entire file processing task is completed.

[0114] Through the above steps, the transfer and processing of files between different operators are realized, making the file processing process more orderly and efficient. The above steps ensure that each operator can operate on the appropriate files through the regeneration of folders and the reclassification of files, avoiding chaos and omission in the file processing process. In addition, the above steps make full use of the functions of different operators to process files at multiple levels and stages, ensuring the integrity and coherence of the entire file processing flow, and at the same time improving the flexibility and scalability of the file processing system.

[0115] An embodiment of the present application provides an operator-based file type screening device. The operator-based file type screening device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the operator-based file type screening method in the first embodiment above.

[0116] Next, refer to Figure 5 , which shows a schematic structural diagram of an operator-based file type screening device suitable for implementing the embodiments of the present application. The operator-based file type screening device in the embodiments of the present application may include various hardware and software components for implementing the operator-based file type screening method. Figure 5 The shown operator-based file type screening device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0117] As Figure 5As shown, the operator-based file type screening device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the operator-based file type screening device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the operator-based file type screening device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an operator-based file type screening device with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems can be implemented or had.

[0118] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowchart can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the method of the embodiments disclosed in the present application are executed.

[0119] The operator-based file type screening device provided in the present application adopts the operator-based file type screening method in the above embodiments, and can solve the technical problem that in corpus processing, the files are not screened and classified, resulting in the operator performing ineffective processing on inapplicable file types and unable to generate useful result files. Compared with the prior art, the beneficial effects of the operator-based file type screening device provided in the present application are the same as those of the operator-based file type screening method provided in the above embodiments, and other technical features in the operator-based file type screening device are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0120] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0121] As mentioned above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0122] An embodiment of this application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the operator-based file type screening method in the above embodiments.

[0123] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0124] The above computer-readable storage medium can be included in the operator-based file type screening device; or it can exist separately without being assembled into the operator-based file type screening device.

[0125] The above computer-readable storage medium carries one or more programs, which, when executed by an operator-based file type screening device, cause the operator-based file type screening device to: generate a folder according to the operator instance identifier of the current operator, and generate a source folder, a subsequent source folder, and a target folder under the folder; obtain the operator adaptation type of the current operator, classify the set of files to be processed according to whether the file type matches the operator adaptation type, classify the adapted files into the source folder, and classify the unadapted files into the subsequent source folder; when executing the current operator, obtain the files to be executed from the source folder, and store the processed files in the target folder; after the current operator is executed, continue to execute the next operator until the processing flow of all operators is completed.

[0126] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through an Internet service provider using the Internet).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0128] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0129] The readable storage medium provided in the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned operator-based file type screening method, which can solve the technical problem that in corpus processing, files are not screened and classified, resulting in ineffective processing of inapplicable file types by the operator and unable to generate useful result files. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as those of the operator-based file type screening method provided in the above embodiments, and will not be elaborated here.

[0130] The embodiments of the present application provide a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the operator-based file type screening method as described above are implemented.

[0131] The computer program product provided in the present application can solve the technical problem that in corpus processing, files are not screened and classified, resulting in ineffective processing of inapplicable file types by the operator and unable to generate useful result files. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present application are the same as those of the operator-based file type screening method provided in the above embodiments, and will not be elaborated here.

[0132] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be included in the patent scope of the present application by the same token.

[0133] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including the element.

[0134] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0135] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

Claims

1. A file type screening method based on an operator, characterized in that: The operator-based file type screening method includes: Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, subsequent source folders, and a target folder under the folder; Obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder; When executing the current operator, the file to be executed is obtained from the source folder, and the processed file is stored in the target folder; After the current operator is executed, continue to execute the next operator until the processing flow of all operators is completed; After the current operator is executed, the next operator is continued to be executed until the processing flow of all operators is completed, including: After the current operator is executed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the steps of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder; After the step of re-executing the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder and a target folder under the folder when the next operator is executed, the step further includes: Based on the subsequent source folder and the target folder corresponding to the previous operator, re-execute the step of classifying according to whether the file type matches the operator adaptation type; Based on the reclassified source folder, the next operator is executed until all operator processing flows are completed.

2. The operator-based file type screening method according to claim 1, characterized in that: Before the step of obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder, the step includes: Get the file name of each file in the collection of files to be processed; A file suffix of each of the file names is extracted, and a file type of each file is determined based on the file suffix.

3. The operator-based file type screening method according to claim 1, characterized in that: Before the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder and a target folder under the folder, the following steps are included: Receive an operator creation instruction, where the operator creation instruction includes an operator instance identifier and a file type that the operator is adapted to; According to the operator creation instruction, a correspondence between the operator and the file type is stored.

4. The operator-based file type screening method according to claim 1, characterized in that: The step of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder, further includes: Determine the sequence of operators required to be executed on the set of files to be processed according to the processing requirements; According to the operator sequence, a corresponding folder is generated for each operator in turn, and a source folder, a subsequent source folder and a target folder are generated under each folder.

5. The operator-based file type screening method according to claim 1, characterized in that: The step of obtaining the file to be executed from the source folder and storing the processed file in the target folder when executing the current operator includes: When executing the current operator, obtaining the file to be executed from the source folder corresponding to the current operator; Execute the preset processing operation corresponding to the current operator on each file to be executed; Each file after completing the preset processing operation is stored in the target folder corresponding to the current operator.

6. The operator-based file type screening method according to claim 1, characterized in that: The step of obtaining the file to be executed from the source folder and storing the processed file in the target folder when executing the current operator also includes: If the source folder corresponding to the current operator is empty, the execution of the current operator is skipped and the execution process of the next operator is entered.

7. A file type screening device based on an operator, characterized in that: The operator-based file type screening device includes: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the operator-based file type screening method according to any one of claims 1 to 6.

8. A computer storage medium, characterized in that: The computer storage medium is a computer-readable storage medium, and a computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the operator-based file type screening method according to any one of claims 1 to 6 are implemented.

Images