File type screening method and device based on operator and computer storage medium

Abstract

The invention discloses an operator-based file type screening method and device and a computer storage medium, and relates to the technical field of text data processing, the method comprises the following steps: generating a folder according to an operator instance identifier of a current operator, and generating a source folder, a subsequent source folder and a target folder under the folder; obtaining an operator adaptation type of a current operator, classifying the to-be-processed file set according to whether the file type is matched with the operator adaptation type or not, classifying adaptive files to a source folder, and classifying unadaptive files to a subsequent source folder; when the current operator is executed, a file needing to be executed is obtained from the source folder, and the processed file is stored in the target folder; and after the execution of the current operator is completed, continuing to execute the next operator until the processing flow of all operators is completed. According to the method, the corresponding folders are created for the operators, and the files are classified according to the operator adaptation types, so that processing requirements are accurately matched, and resource waste is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of text data processing, and in particular to an operator-based file type screening method, device and computer storage medium. Background Art

[0002] With the development of linguistics-related research and applications such as Natural Language Processing (NLP), text mining, and information retrieval, corpus processing has become increasingly important. Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on large-scale text data sets (i.e., corpora), aiming to extract useful information from large amounts of text data to support a wide range of applications. Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction, and final analysis, involving a variety of file types, such as plain text files, rich text format files, web documents, emails, social media posts, and even non-traditional text data sources such as images and videos.

[0003] In corpus processing, "operator" refers to a reusable processing unit or function that undertakes different tasks such as data reading, conversion, cleaning, feature extraction, etc. There are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc. However, when actually processing corpora containing multiple types of files, the files are often sent directly to the operator for execution without being screened and classified. It should be clear that an operator cannot adapt to all types of files. For example, image operators cannot process text files. Therefore, when an operator encounters an unmatched file type, it is easy for the operator to perform invalid processing on the unmatched file type, which not only fails to produce useful result files, but also causes a waste of resources. Summary of the invention

[0004] The main purpose of this application is to provide an operator-based file type screening method, device and computer storage medium, aiming to solve the technical problem that files are not screened and classified in corpus processing, resulting in invalid processing of incompatible file types by operators and failure to generate useful result files.

[0005] To achieve the above object, an embodiment of the present application provides a file type screening method based on an operator, and the file type screening method based on an operator includes: Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, subsequent source folders, and a target folder under the folder; Obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder; When executing the current operator, the file to be executed is obtained from the source folder, and the processed file is stored in the target folder; After the current operator is executed, the next operator is executed until the processing flow of all operators is completed.

[0006] In one embodiment, before the step of obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder, the method includes: Get the file name of each file in the collection of files to be processed; A file suffix of each of the file names is extracted, and a file type of each file is determined based on the file suffix.

[0007] In one embodiment, before the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder, and a target folder under the folder, the following steps are included: Receive an operator creation instruction, where the operator creation instruction includes an operator instance identifier and a file type that the operator is adapted to; According to the operator creation instruction, a correspondence between the operator and the file type is stored.

[0008] In one embodiment, the step of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder further includes: Determine the sequence of operators required to be executed on the set of files to be processed according to the processing requirements; According to the operator sequence, a corresponding folder is generated for each operator in turn, and a source folder, a subsequent source folder and a target folder are generated under each folder.

[0009] In one embodiment, when executing the current operator, the step of obtaining the file to be executed from the source folder and storing the processed file in the target folder includes: When executing the current operator, obtaining the file to be executed from the source folder corresponding to the current operator; Execute the preset processing operation corresponding to the current operator on each file to be executed; Each file after completing the preset processing operation is stored in the target folder corresponding to the current operator.

[0010] In one embodiment, when executing the current operator, the step of obtaining the file to be executed from the source folder and storing the processed file in the target folder further includes: If the source folder corresponding to the current operator is empty, the execution of the current operator is skipped and the execution process of the next operator is entered.

[0011] In one embodiment, after the current operator is executed, the step of continuing to execute the next operator until the processing flow of all operators is completed includes: After the current operator is executed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the steps of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, subsequent source folders and a target folder under the folder.

[0012] In one embodiment, when the next operator is executed, after re-executing the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder and a target folder under the folder, the method further includes: Based on the subsequent source folder and the target folder corresponding to the previous operator, re-execute the step of classifying according to whether the file type matches the operator adaptation type; Based on the reclassified source folder, the next operator is executed until all operator processing flows are completed.

[0013] An embodiment of the present application also provides an operator-based file type screening device, the operator-based file type screening device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the operator-based file type screening method as described above.

[0014] An embodiment of the present application also provides a computer storage medium, which is a computer-readable storage medium. A computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the operator-based file type screening method described above are implemented.

[0015] The embodiment of the present application discloses a file type screening method based on an operator, which generates a folder according to the operator instance identifier of the current operator, and generates a source folder, a subsequent source folder and a target folder under the folder; obtains the operator adaptation type of the current operator, and classifies the set of files to be processed according to whether the file type matches the operator adaptation type, and classifies the adapted files into the source folder, and the incompatible files into the subsequent source folder; when executing the current operator, obtains the files to be executed from the source folder, and stores the processed files into the target folder; after the current operator is executed, continues to execute the next operator until the processing flow of all operators is completed. The present application creates corresponding folders for operators, and classifies files according to the operator adaptation type, accurately matches processing requirements, and avoids waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a first embodiment of an operator-based file type screening method involved in an embodiment of the present application; Figure 2 This is a flow chart of a second embodiment of an operator-based file type screening method according to an embodiment of the present application; Figure 3 This is a flow chart of a third embodiment of an operator-based file type screening method according to an embodiment of the present application; Figure 4 This is a flow chart of a fourth embodiment of an operator-based file type screening method according to an embodiment of the present application; Figure 5 This is a schematic diagram of the structure of an operator-based file type screening device involved in an embodiment of the present application.

[0017] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] With the development of linguistics-related research and applications such as Natural Language Processing (NLP), text mining, and information retrieval, corpus processing has become increasingly important. Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on large-scale text data sets (i.e., corpora), aiming to extract useful information from large amounts of text data to support a wide range of applications. Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction, and final analysis, involving a variety of file types, such as plain text files, rich text format files, web documents, emails, social media posts, and even non-traditional text data sources such as images and videos.

[0020] In corpus processing, "operator" refers to a reusable processing unit or function that undertakes different tasks such as data reading, conversion, cleaning, feature extraction, etc. There are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc. However, when actually processing corpora containing multiple types of files, the files are often sent directly to the operator for execution without being screened and classified. It should be clear that an operator cannot adapt to all types of files. For example, image operators cannot process text files. Therefore, when an operator encounters an unmatched file type, it is easy for the operator to perform invalid processing on the unmatched file type, which not only fails to produce useful result files, but also causes a waste of resources.

[0021] In order to solve the above defects existing in the related art, the embodiment of the present application proposes a file type screening method based on an operator, which generates a folder according to the operator instance identifier of the current operator, and generates a source folder, a subsequent source folder and a target folder under the folder; obtains the operator adaptation type of the current operator, and classifies the set of files to be processed according to whether the file type matches the operator adaptation type, and classifies the adapted files into the source folder, and the incompatible files into the subsequent source folder; when executing the current operator, obtains the files to be executed from the source folder, and stores the processed files into the target folder; after the current operator is executed, continues to execute the next operator until the processing flow of all operators is completed. The present application creates corresponding folders for operators, and classifies files according to operator adaptation types, accurately matches processing requirements, and avoids waste of resources.

[0022] It should be noted that the execution subject of this embodiment can be an operator-based file type screening system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an operator-based file type screening device capable of realizing the above functions, etc. The following takes the operator-based file type screening system (hereinafter referred to as the "system") as an example to illustrate this embodiment and the following embodiments.

[0023] For the operator-based file type screening method of the first embodiment proposed in this application, please refer to Figure 1 The method comprises steps S10 to S40: Step S10: Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, a subsequent source folder, and a target folder under the folder.

[0024] Corpus processing is a process of performing a series of automated or semi-automated processing and analysis on large-scale text data sets (i.e., corpora). It aims to extract useful information from large amounts of text data to support many applications such as natural language processing (NLP), text mining, information retrieval, and other linguistic research.

[0025] Corpus processing includes multiple stages such as data acquisition, preprocessing, feature extraction and final analysis, and each stage requires corresponding technical means and tools.

[0026] In practice, corpus processing involves a variety of file types, such as plain text files (such as ".txt"), rich text format files (such as ".docx", ".pdf"), web documents, emails, social media posts, and even non-traditional text data sources such as images and videos. During the processing, different file types require different operators to complete specific tasks, such as text cleaning, word segmentation, part-of-speech tagging, named entity recognition, etc.

[0027] In addition, with the increasing amount of multimedia content, corpus processing also includes processing of non-traditional text data sources such as processing of images through optical character recognition (OCR) and processing of videos through video frame conversion and in-frame text recognition.

[0028] It should be noted that in corpus processing, "operators" refer to reusable processing units or functions that undertake different tasks such as data reading, data conversion, data cleaning, feature extraction, analysis and processing, and result output. Operators can be regarded as a node or module in the data processing process, and each node has a specific processing task.

[0029] When processing multimedia data sets, the concept of "operator" can be extended to specific processing steps or functional modules for different types of files, that is, there are corresponding operators for different file types, such as text cleaning operators, PDF text extraction operators, OCR operators, video frame conversion operators, etc.

[0030] Exemplarily, in the text file processing operator, the function of the text cleaning operator is to remove non-text content (such as HTML tags) in the text and normalize the text format. The input is a text file, and the output is clean text content. In the PDF file processing operator, the function of the PDF text extraction operator is to extract text content from a PDF file. The input is a PDF file, and the output is the text content in the PDF file. In the image file processing operator, the function of the OCR operator is to convert the text in the image into text format. The input is an image file, and the output is the text content in the extracted image file. In the video file processing operator, the function of the video frame conversion operator is to convert a video file into a series of image frames. The input is a video file, and the input is a sequence of image frames.

[0031] Different operators have different requirements for file types. For example, image file processing operators cannot process text files. If the collection of files to be processed is not filtered and classified, but sent directly to the operator for execution, incompatible files and operators cannot produce useful result files, but the operator will still execute, resulting in a waste of resources and prone to errors, leading to unstable services. In this embodiment, when each operator is executed, an operator instance is generated, and the operator instance identifier can be generated using the snowflake algorithm. The snowflake algorithm is an algorithm for generating a unique identifier, which can ensure that each operator instance has a unique identifier in the entire system to avoid confusion and conflict during the processing process.

[0032] When an operator starts executing, the system will create a folder based on the operator instance identifier. This folder is the basic storage unit for each subsequent operator to process the collection of files to be processed. Under this folder, three subfolders will be further generated, namely the source folder (source), the subsequent source folder (nextSource), and the target folder (target).

[0033] The source folder is used to store files that the current operator can process. It is the starting point for the operator to start processing files and contains files to be processed that are compatible with the current operator. The subsequent source folder is used to store files that are outside the processing scope of the current operator (that is, the operator adaptation type that is not compatible with the current operator) but may be used later in the entire corpus processing process, providing temporary storage or backup space for processing operations at different stages. The target folder is used to store the processing results of the current operator. After the current operator completes the processing operation on the file, the final processing result will be stored in the target folder for subsequent viewing, analysis or further use.

[0034] The above method of generating corresponding folders and subfolders for each operator ensures that in a complex corpus processing process, different types of files can be processed and stored in an orderly manner under the processing of each operator. This makes the file processing processes between different operators independent of each other and avoids file confusion between different operator operations. In addition, it also helps to reasonably allocate and manage file resources. By storing files according to the processing stage, it is easy to find and use files during the processing process, which improves the stability and operability of the entire corpus processing system.

[0035] Step S20: obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder.

[0036] In this embodiment, the set of files to be processed is a set of files that need to be processed, which may include a variety of different file types. The operator adaptation type defines the range of file types that the current operator can process. When a user creates an operator, it is predetermined which file type the operator is adapted to. Therefore, when a user creates an operator, the operator adaptation type of the operator is identified by specifying the label of the operator. According to the label of the operator, the operator adaptation type of the current operator can be obtained.

[0037] Exemplarily, the text data cleaning operator is suitable for file types such as "jsonl", "json", "txt", "csv", "xlsx", and "xls". Then the name of the operator can be expressed as text data cleaning, the label type is CLEANTXT, and the operator adaptation types of the operator include "jsonl", "json", "txt", "csv", "xlsx", and "xls". When the current operator is text data cleaning, if a file in the set of files to be processed is of image type, then according to the operator adaptation type of the current operator, it can be known that the file of the image type is not suitable for the current operator, and it is placed in the subsequent source folder under the operator.

[0038] In this embodiment, the three subfolders of the source folder, the subsequent source folder and the target folder are used to store files obtained by classifying the to-be-processed file set according to file types and result files processed by the operator.

[0039] In an optional embodiment, step S20 includes steps S201-S202 before step S20: Step S201: Obtain the file name of each file in the set of files to be processed.

[0040] Step S202: extracting the file suffix of each file name, and determining the file type of each file based on the file suffix.

[0041] In this embodiment, the file type is obtained according to the file extension or file header information.

[0042] Specifically, the file operation library is used to read part of the content of each file in the set of files to be processed, and the file type is determined based on its format characteristics. For common file types (such as ".txt", ".jpg", ".pdf", etc.), the file extension can also be used for judgment. The file type of the file is determined by extracting the file suffix in the file name.

[0043] Then, each file in the set of files to be processed is compared with the operator adaptation type of the current operator being executed to determine whether they match, and each file in the set of files to be processed is classified.

[0044] When classifying each file in the set of files to be processed, you can use the file operation function to move or copy the file to the corresponding folder. Taking Python as an example, you can use the copy or move function in the shutil module. For files that are compatible with the operator adaptation type of the current operator, copy or move the file to the source folder; for files that are not compatible, copy or move the file to the subsequent source folder.

[0045] Through the above steps, the files in the to-be-processed file set will be stored in different subfolders according to the matching of the file types, which provides a clear file classification and storage structure for subsequent operator execution and processing, and helps to improve the manageability and efficiency of corpus processing.

[0046] Step S30: when executing the current operator, the files to be executed are obtained from the source folder, and the processed files are stored in the target folder.

[0047] In this embodiment, after the to-be-processed file set is classified, the current operator is executed. In the source folder corresponding to the current operator, the files classified as matching the operator adaptation type of the current operator in the above steps are stored, and these files are the objects that the current operator needs to process.

[0048] Specifically, first traverse the files in the source folder, for each file, open it in read mode and read the file content, then use the current operator to process these files. The specific processing operation depends on the specific function and implementation method of the current operator. Finally, store the processed results in the target folder.

[0049] For example, if the current operator is a data cleaning operator, the data in the file will be cleaned. If the current operator is a format conversion operator, the format of the file will be converted. After the current operator completes the file processing, the processed file will be stored in the target folder. Among them, the processed file can be stored using the file operation library, such as using the open function in Python to open the file in the target folder in write mode, and then using the write method to write the processed file content to the file.

[0050] It should be noted that using the open function to open a file in write mode is to create or open a file and write the processed content into it.

[0051] Step S40: After the current operator is executed, continue to execute the next operator until the processing flow of all operators is completed.

[0052] In this embodiment, the set of files to be processed usually includes files of multiple different file types. Therefore, in the process of corpus processing, due to the diversity of file types, multiple operators are usually required to process the set of files to be processed. Each operator has a specific function and together completes a complex file processing task. These operators will be executed in sequence according to a preset order to ensure the orderliness and coherence of the entire file processing process.

[0053] The current operator refers to the operator that is currently executing an operation. After the current operator is completed, the system will continue to execute the next operator until all operators are completed, thus completing the entire corpus processing flow of the set of files to be processed.

[0054] In an optional implementation, a queue or list is created to set all operators that need to be executed in the corpus processing flow. Then, a scheduler or other scheduling mechanism is used to execute the corresponding operators in the order of the queue or list. During the execution of the operator, the execution status of each operator is tracked so that after the current operator is executed, it can be determined whether there are other operators that need to be executed.

[0055] Please refer to Figure 2 The operator-based file type screening method of the second embodiment proposed in this application includes steps S101 to S102 before step S10: Step S101: receiving an operator creation instruction, wherein the operator creation instruction includes an operator instance identifier and a file type compatible with the operator.

[0056] Step S102: According to the operator creation instruction, the correspondence between the operator and the file type is stored.

[0057] When processing corpus, it is necessary to ensure that each file can be processed correctly. Therefore, users will configure corresponding operators for each file type. When users create operators, they identify the operator adaptation type of the corresponding operator by specifying the operator's label.

[0058] When the system receives an operator creation instruction, it needs to store the correspondence between the operator and the file type that the operator is adapted to. When the operator is subsequently executed, the system can determine the range of files processed by the current operator based on the operator's label through the mapping relationship, avoiding incorrect matching of operators and file types during the processing process, and ensuring the accuracy and efficiency of file processing. In addition, when it is necessary to add or modify an operator, you only need to update the stored correspondence to integrate the new operator into the entire corpus processing process, providing the necessary information basis for the orderly screening and processing of subsequent files.

[0059] Please refer to Figure 3 In the operator-based file type screening method of the third embodiment proposed in the present application, step S10 further includes steps S110 to S120: Step S110: Determine the operator sequence required to be executed for the set of files to be processed according to the processing requirements.

[0060] In this embodiment, the processing requirement is only a specific processing target for the set of files to be processed. When processing the set of files to be processed, different file types and different processing requirements mean that different operators need to be used. For example, extracting text information from a picture may need to be passed through an OCR operator first, while for a video, it may be necessary to first convert it into a series of frame images and then process the text content in each frame separately.

[0061] Therefore, reasonable operator arrangement can ensure that each file is properly processed during the processing process, so as to achieve the expected processing effect. Operator arrangement is the orderly arrangement and scheduling of operators required to be executed for a file. For a set of files to be processed, the user clearly knows the processing effect they want to achieve, so which operator to execute first and which operator to execute later can be determined according to the processing requirements.

[0062] These operators are arranged in the order of processing requirements to obtain the operator sequence.

[0063] Step S120: Generate a corresponding folder for each operator in turn according to the operator sequence, and generate a source folder, a subsequent source folder and a target folder under each folder.

[0064] Since different operators need independent spaces to store various files in the processing process when processing files, in order to ensure the orderliness and independence of the operations, in this embodiment, a corresponding folder is generated for each operator, so that the folder becomes the basic storage unit for the operator to process files. Under this basic storage unit, source folders, subsequent source folders, and target folders are further generated.

[0065] In this embodiment, in order to meet the diverse processing requirements in complex file processing tasks, the processing process needs to be decomposed into multiple tasks and executed in sequence by different operators. The operation of each operator requires an independent storage area to avoid file confusion and operation interference between different operators.

[0066] In the above way, each operator is assigned independent folders and subfolders, which helps to organize files clearly, facilitates the management and monitoring of the processing process of each operator, and improves the efficiency and accuracy of file processing. At the same time, the design of the above storage structure also facilitates the distinction and preservation of the processing results of different operators, helps to ensure the independence and integrity of files at different processing stages, provides a good foundation for subsequent file screening, processing and storage operations, and meets the file processing tasks under different processing requirements.

[0067] Please refer to Figure 4 In the operator-based file type screening method of the fourth embodiment proposed in the present application, step S30 includes steps S310 to S330: Step S310: when executing the current operator, obtain the file to be executed from the source folder corresponding to the current operator.

[0068] In this embodiment, when the current operator is executed, the file to be executed is obtained from the source folder corresponding to the current operator. The source folder here refers to the file created for the current operator when the current operator is executed, which stores files adapted to the operator adaptation type of the current operator. These files are files that meet the processing range of the current operator after classification in the above steps, and provide a source of input data for the operation of the current operator.

[0069] Step S320: Execute the preset processing operation corresponding to the current operator on each file to be executed.

[0070] In this embodiment, the preset processing operation is an operation pre-set according to the function and task of the current operator. For example, for a text file, the preset processing operation may be data cleaning, format conversion, or information extraction. For an image file, it may be image enhancement, size adjustment, or format conversion. Each operator has its own specific preset processing operation, which is intended to perform specific processing on the file to meet different processing requirements.

[0071] Step S330: storing each file after completing the preset processing operation into the target folder corresponding to the current operator.

[0072] The target folder is used to store the result files after the current operator has processed them. The purpose of storing the processed files in the target folder is, on the one hand, to centrally manage the processing results for easy subsequent viewing and use; on the other hand, it also helps to maintain the orderliness of the entire file processing process and avoid confusion between the processing results and other operators or operations.

[0073] Through the above method, each operator has a corresponding file processing path, obtains the input file from the source folder, performs its own processing operation, and finally stores the result in the target folder.

[0074] Furthermore, step S30 further includes step S340: Step S340: If the source folder corresponding to the current operator is empty, skip the execution of the current operator and enter the execution process of the next operator.

[0075] In this embodiment, if an operator that has no files to process is forced to perform an operation, it will not only waste system resources, but may also cause errors or abnormal situations. Therefore, if it is found that the source folder corresponding to the current operator is empty, the execution of the current operator is skipped and the execution process of the next operator is directly entered. In this way, the entire file processing process can be made more efficient and smooth, ensuring that the operator only operates on situations where there are actual files to be processed, and for operators that have no files to process, the processing flow is directly transferred to the next operator, ensuring the consistency and effectiveness of the entire file processing process.

[0076] In the fifth embodiment of the file type screening method based on operators proposed in the present application, step S40 includes step S410: Step S410: After the current operator is executed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the steps of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, subsequent source folders and a target folder under the folder.

[0077] When executing corpus processing or file processing flow, the operator sequence will be pre-set to determine the order of processing of each operator in the processing process. If the current operator is executed (including the source folder corresponding to the current operator is empty, skipping the execution of the current operator), it is necessary to check whether there are subsequent operators that need to continue processing files. If there is a next operator, then continue the execution of the next operator. When the next operator is executed, the operation of generating folders and corresponding subfolders (source folders, subsequent source folders, and target folders) according to the operator instance identifier will be re-executed. This is to provide an independent file storage and operation environment for the next operator to ensure the orderliness and independence of the next operator processing process.

[0078] Furthermore, after step S410, steps S420 to S430 are also included: Step S420: Based on the subsequent source folder and the target folder corresponding to the previous operator, the step of classifying according to whether the file type matches the operator adaptation type is re-executed.

[0079] In this embodiment, the subsequent source folder of the previous operator stores files that failed to adapt to the previously executed operator in the previous operation, and the target folder of the previous operator stores the result files processed by the previous operator.

[0080] After the current operator is executed, the next operator is executed according to the operator sequence. When the next operator is executed, the file type and operator adaptation type need to be reclassified, and the files in the subsequent source folder and target folder of the previous operator are filtered again according to the adaptation type of the new operator (that is, the next operator) to prepare the appropriate file source for the operation of the next operator. Then the classification results are saved again in the subfolder corresponding to the new operator.

[0081] This embodiment ensures the consistency and adaptability of file processing between different operators, so that files that were not processed by the previous operator and the processed files can be reasonably processed and distributed according to the adaptability of the new operator.

[0082] Step S430: Based on the reclassified source folder, continue to execute the next operator until all operator processing flows are completed.

[0083] This embodiment is a mechanism for advancing the file processing flow, which ensures that the entire file processing process can be carried out in an orderly manner according to the operator sequence. After processing a file, each operator can pass the processed file and the unprocessed file to the next operator, and reasonably redistribute and process the file resources until the entire file processing task is completed.

[0084] Through the above steps, the transfer and processing of files between different operators are realized, making the file processing process more orderly and efficient. The above steps ensure that each operator can operate on the basis of appropriate files by regenerating folders and reclassifying files, avoiding confusion and omissions in the file processing process. In addition, the above steps make full use of the functions of different operators to process files at multiple levels and stages, ensuring the integrity and coherence of the entire file processing process, while also improving the flexibility and scalability of the file processing system.

[0085] An embodiment of the present application provides an operator-based file type screening device, and the operator-based file type screening device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the operator-based file type screening method in the above-mentioned embodiment 1.

[0086] Reference below Figure 5 , which shows a schematic diagram of the structure of an operator-based file type screening device suitable for implementing an embodiment of the present application. The operator-based file type screening device in the embodiment of the present application may include various hardware and software components for implementing an operator-based file type screening method. Figure 5 The operator-based file type screening device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0087] like Figure 5As shown, the file type screening device based on the operator may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the file type screening device based on the operator are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the operator-based file type screening device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an operator-based file type screening device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0088] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0089] The operator-based file type screening device provided by the present application adopts the operator-based file type screening method in the above embodiment, which can solve the technical problem that the files are not screened and classified in the corpus processing, resulting in the operator performing invalid processing on the incompatible file type and failing to generate useful result files. Compared with the prior art, the beneficial effects of the operator-based file type screening device provided by the present application are the same as the beneficial effects of the operator-based file type screening method provided by the above embodiment, and the other technical features of the operator-based file type screening device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0090] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0091] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0092] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the operator-based file type screening method in the above embodiment.

[0093] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0094] The computer-readable storage medium may be included in the operator-based file type screening device; or may exist independently without being assembled into the operator-based file type screening device.

[0095] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an operator-based file type screening device, the operator-based file type screening device: generates a folder according to the operator instance identifier of the current operator, and generates a source folder, a subsequent source folder and a target folder under the folder; obtains the operator adaptation type of the current operator, and classifies the set of files to be processed according to whether the file type matches the operator adaptation type, classifies the adapted files into the source folder, and classifies the incompatible files into the subsequent source folder; when executing the current operator, obtains the files to be executed from the source folder, and stores the processed files into the target folder; after the execution of the current operator is completed, continues to execute the next operator until the processing flow of all operators is completed.

[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0099] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned operator-based file type screening method, and can solve the technical problem that the files are not screened and classified in the corpus processing, resulting in the operator performing invalid processing on the incompatible file type and failing to generate useful result files. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the operator-based file type screening method provided by the above-mentioned embodiment, and will not be elaborated here.

[0100] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned operator-based file type screening method.

[0101] The computer program product provided by the present application can solve the technical problem that the files are not screened and classified in the corpus processing, resulting in the operator performing invalid processing on the incompatible file type and failing to generate useful result files. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the file type screening method based on the operator provided by the above embodiment, and will not be elaborated here.

[0102] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

[0103] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0105] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A file type screening method based on an operator, characterized in that: The operator-based file type screening method includes: Generate a folder according to the operator instance identifier of the current operator, and generate a source folder, subsequent source folders, and a target folder under the folder; Obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder; When executing the current operator, the file to be executed is obtained from the source folder, and the processed file is stored in the target folder; After the current operator is executed, the next operator is executed until the processing flow of all operators is completed.

2. The operator-based file type screening method according to claim 1, characterized in that: Before the step of obtaining the operator adaptation type of the current operator, classifying the to-be-processed file set according to whether the file type matches the operator adaptation type, classifying the adapted files into the source folder, and classifying the incompatible files into the subsequent source folder, the step includes: Get the file name of each file in the collection of files to be processed; A file suffix of each of the file names is extracted, and a file type of each file is determined based on the file suffix.

3. The operator-based file type screening method according to claim 1, characterized in that: Before the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder and a target folder under the folder, the following steps are included: Receive an operator creation instruction, where the operator creation instruction includes an operator instance identifier and a file type that the operator is adapted to; According to the operator creation instruction, a correspondence between the operator and the file type is stored.

4. The operator-based file type screening method according to claim 1, characterized in that: The step of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, a subsequent source folder, and a target folder under the folder, further includes: Determine the sequence of operators required to be executed on the set of files to be processed according to the processing requirements; According to the operator sequence, a corresponding folder is generated for each operator in turn, and a source folder, a subsequent source folder and a target folder are generated under each folder.

5. The operator-based file type screening method according to claim 1, characterized in that: The step of obtaining the file to be executed from the source folder and storing the processed file in the target folder when executing the current operator includes: When executing the current operator, obtaining the file to be executed from the source folder corresponding to the current operator; Execute the preset processing operation corresponding to the current operator on each file to be executed; Each file after completing the preset processing operation is stored in the target folder corresponding to the current operator.

6. The operator-based file type screening method according to claim 1, characterized in that: The step of obtaining the file to be executed from the source folder and storing the processed file in the target folder when executing the current operator also includes: If the source folder corresponding to the current operator is empty, the execution of the current operator is skipped and the execution process of the next operator is entered.

7. The operator-based file type screening method according to claim 1, characterized in that: After the current operator is executed, the next operator is continued to be executed until the processing flow of all operators is completed, including: After the current operator is executed or skipped, determine whether there is an execution process for the next operator according to the operator sequence. If so, when the next operator is executed, re-execute the steps of generating a folder according to the operator instance identifier of the current operator, and generating a source folder, subsequent source folders and a target folder under the folder.

8. The operator-based file type screening method according to claim 7, characterized in that: After the step of re-executing the step of generating a folder according to the operator instance identifier of the current operator and generating a source folder, a subsequent source folder and a target folder under the folder when the next operator is executed, the step further includes: Based on the subsequent source folder and the target folder corresponding to the previous operator, re-execute the step of classifying according to whether the file type matches the operator adaptation type; Based on the reclassified source folder, the next operator is executed until all operator processing flows are completed.

9. A file type screening device based on an operator, characterized in that: The operator-based file type screening device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the operator-based file type screening method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is a computer-readable storage medium, and a computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the operator-based file type screening method according to any one of claims 1 to 8 are implemented.

Images