File processing method, device and equipment and readable storage medium
By obtaining target configuration information from the database and converting files into standardized formats, the problem of inefficient processing of large files in the financial market is solved, and efficient file processing and business needs are achieved.
Patent Information
- Application Number
- CN202411715696.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-18
AI Technical Summary
The existing technology has inefficient management and disposal of large documents in the financial markets and cannot meet the growing business needs.
By obtaining the target configuration information of the pending files from the database, the pending files are converted into target file data in a standardized format and sent to the message queue.
It improves the processing efficiency of file processing, can quickly respond to and process large files, and meet business needs.
Smart Images
Figure CN119645946A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and more specifically, to a file processing method, device, equipment and readable storage medium. Background Art
[0002] With the rapid development of the digital age, the number of scenarios for obtaining non-real-time transaction data, financial market information, and valuation information has increased dramatically in the process of building systems in the financial market field. The management and disposal of large files has become an increasingly important and challenging task. However, the current file processing methods are inefficient and cannot meet the growing business needs. Summary of the invention
[0003] The embodiments of the present application provide a file processing method, apparatus, device and readable storage medium, which can improve the processing efficiency of file processing.
[0004] In view of this, an embodiment of the present application provides a file processing method, the method comprising:
[0005] Identify files to be processed;
[0006] Acquire target configuration information of the file to be processed from a database;
[0007] Converting the to-be-processed file into target file data based on the target configuration information, wherein the format of the target file data is a standardized format;
[0008] The target file data is sent to a message queue.
[0009] Optionally, converting the to-be-processed file into target file data based on the target configuration information includes:
[0010] Read the file data of the file to be processed line by line;
[0011] Performing information segmentation on the file data to obtain file data to be converted;
[0012] The to-be-converted file data is converted into the target file data based on the target field mapping information in the target configuration information.
[0013] Optionally, before reading the file data of the to-be-processed file line by line, the method further includes:
[0014] The to-be-processed file is decompressed based on the target decompression information in the target configuration information.
[0015] Optionally, the acquiring target configuration information of the to-be-processed file from a database includes:
[0016] Add a distributed lock to the file to be processed;
[0017] The target configuration information is acquired from the database based on the identification information of the file to be processed.
[0018] Optionally, determining the file to be processed includes:
[0019] Determine the file path to be monitored according to the file configuration set;
[0020] Based on the file listener, monitor whether there is a file change in the file path to be monitored;
[0021] If there is a file change in the file path to be monitored, the changed file is determined as a file to be processed.
[0022] Optionally, the message queue is a Kafka message queue.
[0023] The present application also provides a file processing device, the device comprising:
[0024] A determination unit, used for determining a file to be processed;
[0025] An acquisition unit, used for acquiring target configuration information of the file to be processed from a database;
[0026] A conversion unit, configured to convert the to-be-processed file into target file data based on the target configuration information, wherein the format of the target file data is a standardized format;
[0027] The sending unit is used to send the target file data to the message queue.
[0028] Optionally, the conversion unit is specifically used to:
[0029] Read the file data of the file to be processed line by line;
[0030] Performing information segmentation on the file data to obtain file data to be converted;
[0031] The to-be-converted file data is converted into the target file data based on the target field mapping information in the target configuration information.
[0032] The embodiment of the present application also provides a computer device, including: a memory, a processor, and a bus system;
[0033] Wherein, the memory is used to store programs;
[0034] The processor is used to execute the program in the memory to implement any one of the file processing methods described above;
[0035] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
[0036] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute any one of the file processing methods described above.
[0037] The embodiment of the present application provides a file processing method, including: determining a file to be processed; obtaining target configuration information of the file to be processed from a database; converting the file to be processed into target file data based on the target configuration information, wherein the format of the target file data is a standardized format; and sending the target file data to a message queue. It can be seen that the present application can obtain target configuration information pre-configured for the file to be processed from a database, and can quickly convert the file to be processed into target file data in a standardized format based on the target configuration information, and finally send the converted target file data to a message queue for further processing, thereby improving the processing efficiency of file processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0039] Figure 1 A flowchart of a file processing method provided in an embodiment of the present application;
[0040] Figure 2 A schematic diagram of the structure of a file processing model provided in an embodiment of the present application;
[0041] Figure 3 A flowchart of another file processing method provided in an embodiment of the present application;
[0042] Figure 4 A schematic diagram of the structure of a file processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0045] First, some of the terms that may be involved in this application are explained:
[0046] File monitor: It is a processor that can monitor file change events in the system directory. It monitors file operations such as creation, modification, and deletion. It is a processing method that helps improve the flexibility and automation level of system file operations.
[0047] Distributed lock: Also known as Redis lock, it is a distributed lock mechanism based on Redis. Given Redis's transaction processing capabilities and data structure characteristics, distributed locks can achieve mutually exclusive access or synchronous operations on multiple nodes, and are easy to use, high-performance, and highly scalable.
[0048] Java stream processing: refers to using the stream API in Java to perform a series of transformation and processing operations on data. Through Java stream processing, data is filtered, mapped, and reduced to achieve data cleaning, transformation, and analysis. The concise syntax and powerful functions make Java stream processing have high data processing efficiency and strong code readability.
[0049] Kafka message queue: a distributed stream processing platform used to build real-time data stream pipelines and applications. It provides a high-throughput distributed message subscription system and is a real-time processing message queue with reliable data transmission.
[0050] With the rapid development of the digital age, the number of scenarios for obtaining non-real-time transaction data, financial market information, and valuation information has increased dramatically in the process of building systems in the financial market field. The management and disposal of large files has become an increasingly important and challenging task. However, when processing files, technical personnel are usually required to configure the file processing flow and configuration information in real time. The file processing efficiency is low and cannot meet the growing business needs.
[0051] Therefore, in order to solve the above problems, the embodiments of the present application provide a file processing method, device, equipment and readable storage medium, which can improve the processing efficiency of file processing.
[0052] See also Figure 1 , a file processing method provided in an embodiment of the present application includes the following steps.
[0053] S101, determining files to be processed.
[0054] In the embodiment of the present application, the to-be-processed files that need to be processed may be predetermined, wherein the to-be-processed files may be files that have changed, for example, the to-be-processed files may be newly created files or files that have undergone modification operations.
[0055] In a possible implementation method of the embodiment of the present application, determining the file to be processed includes: determining the file path to be monitored according to the file configuration set; monitoring whether there is a file change in the file path to be monitored based on the file listener; if there is a file change in the file path to be monitored, determining the changed file as the file to be processed. In the embodiment of the present application, the file configuration set can be searched first, and specifically the file monitoring configuration can be read from the file monitoring entity to generate a file monitoring configuration information set; then the file path information to be monitored can be extracted based on the file configuration set, and the file path information to be monitored can be deduplicated to obtain the file path to be monitored. After determining the file path to be monitored, a file listener can be created and started to monitor the file path to be monitored to determine whether there is a file change in the file path to be monitored, that is, to determine whether there is a newly created file or a file that has just been modified. The newly created file or the file that has just been modified can be determined as the file to be processed, so that the files to be processed that need to be processed can be processed in time, further improving the processing efficiency of file processing.
[0056] S102: Obtain target configuration information of the file to be processed from a database.
[0057] In an embodiment of the present application, the configuration information required in the processing of multiple files can be configured in advance, and the correspondence between each file and the configuration information and the configuration information corresponding to each file can be stored in a database, wherein the configuration information can include decompression information and field mapping information, etc. The decompression information can be used to indicate the use of a corresponding decompression tool to decompress the file. If the file is in text format, there is no need to decompress it, and the relevant configuration can be set to empty. The field mapping information can be used to indicate the field mapping relationship between the current format of the file and the standardized format of the file, and the field mapping information can record the way in which the field name corresponding to each position in the file is mapped to the standardized format. After determining the file to be processed, the target configuration information corresponding to the file to be processed can be obtained from the database, so that the file to be processed can be quickly converted into the target file data in a standardized format based on the target configuration information, and finally the converted target file data is sent to the message queue for further processing, thereby improving the processing efficiency of file processing.
[0058] In a possible implementation method of an embodiment of the present application, the target configuration information of the file to be processed is obtained from a database, including: adding a distributed lock to the file to be processed; obtaining the target configuration information from the database based on the identification information of the file to be processed. In an embodiment of the present application, after determining the file to be processed, the lock can be grabbed first, that is, a distributed lock can be added to the file to be processed first to ensure the uniqueness of the event processing, thereby avoiding the repeated processing of the file to be processed. Then, the identification information of the file to be processed can be matched with the regular expression pre-stored in the database to determine the target configuration information corresponding to the file to be processed, where the identification information of the file to be processed can be the file name.
[0059] In addition, it should be noted that before obtaining the target configuration information from the database based on the identification information of the file to be processed, the integrity of the file to be processed can also be verified to ensure the data accuracy of the file to be processed. After the integrity of the file to be processed is verified, the file to be processed can be copied to a specified location for subsequent operations. After obtaining the target configuration information from the database based on the identification information of the file to be processed, the source file of the file to be processed can also be deleted to prevent other processes from repeatedly processing the same file. And release the distributed lock in the file to be processed to ensure that the file processing flow can be triggered again when the same file name appears again.
[0060] S103: Convert the file to be processed into target file data based on the target configuration information.
[0061] The target file data is in a standardized format.
[0062] In an embodiment of the present application, after obtaining the target configuration information of the file to be processed, the file to be processed can be quickly converted into target file data in a standardized format based on the target configuration information of the file to be processed, so that the converted target file data can be sent to a message queue for further processing, thereby improving the processing efficiency of file processing. The standardized format can specifically be a value object (VO). VO represents an object without a unique identifier, and its equality is based on value rather than identity, which means that two VO data are considered equal if they contain the same value.
[0063] In a possible implementation of the embodiment of the present application, the file to be processed is converted into target file data based on the target configuration information, including: reading the file data of the file to be processed line by line; performing information segmentation on the file data to obtain the file data to be converted; and converting the file data to be converted into the target file data based on the target field mapping information in the target configuration information. In the embodiment of the present application, the file data of the file to be processed can first be loaded into the cache line by line by reading line by line, and the cache will be in the form of List<strong> <string>Each line of information is stored in the form of. Then the file data of the file to be processed can be cut to obtain the file data to be converted. The cutting process supports general processes and special processes. For files to be processed in general formats, the preset cutting method can be called directly; for situations that require special processing, the coder needs to pre-write and store the corresponding cutting method. When it is necessary to cut the information of the file to be processed, the information cutting function is completed by obtaining the pre-written cutting method. The default processed file format can be "message1|!message2|!message3", where "|!" is used as an information separator. In the general method, information in this format can be converted into a list form, such as "{"message1", "message2", "message3"}". Finally, the file data to be converted can be converted into target file data with a standardized format based on the target field mapping information. Specifically, the List information can be converted into the standard Json format based on the target field mapping information. For example, "{"message1", "message2", "message3"}" can be converted into "{"key1":"message1", "key2":"message2", "key3":"message3"}", and then the Json information can be converted into VO through the reverse transcription method, that is, "{"key1":"message1", "key2":"message2", "key3":"message3"}", which is converted into a value object with the format of "{key1, key2, key3}".
[0064] In a possible implementation of an embodiment of the present application, before reading the file data of the file to be processed line by line, the method further includes: decompressing the file to be processed based on the target decompression information in the target configuration information. In an embodiment of the present application, the file to be processed can be first decompressed based on the target decompression information in the target configuration information. Specifically, whether the file to be processed needs to be decompressed and the decompression tool required for decompressing the file to be processed can be determined based on the target decompression information. After determining the decompression tool for the file to be processed, the decompression tool can be called to decompress the file to be processed to ensure that subsequent processing can be targeted at the actual content of the file to be processed.
[0065] S104: Send the target file data to the message queue.
[0066] In an embodiment of the present application, after the file to be processed is converted into target file data in a standardized format, the target file data can be sent to a message queue so that the message subscriber can further process or consume the message. Common consumption methods include warehousing data information, displaying file information, etc.
[0067] In one possible implementation of the embodiment of the present application, the message queue is a Kafka message queue. In the embodiment of the present application, the target file data can be sent to the Kafka message queue, so that the standardized data in the Kafka queue can be effectively consumed and utilized, and the diversified needs of different microservices in data processing can be met, thereby improving the flexibility and scalability of the file processing method.
[0068] It can be seen that a file processing method is provided in an embodiment of the present application, which can obtain target configuration information pre-configured for the file to be processed from a database, and can quickly convert the file to be processed into target file data in a standardized format based on the target configuration information, and finally send the converted target file data to a message queue for further processing, thereby improving the processing efficiency of file processing.
[0069] For ease of understanding, the embodiments of the present application will introduce a file processing method provided by the present application in combination with actual application scenarios.
[0070] See also Figure 2 , the embodiment of the present application provides a file processing model, specifically including:
[0071] 1. File Monitor Model
[0072] The file monitor model is a model specifically designed to monitor file arrival. Its core function is to monitor the arrival of files in the system directory in real time to ensure immediate discovery and response of files. When a file arrives, the model can trigger subsequent file processing processes, which may include decompression, formatting, classification, and other specific processing based on preset or configured methods. In addition, the file monitor model also records file processing life cycle information, such as the file's arrival path, differentiated processing methods, etc.
[0073] In order to enhance user-friendliness, the file monitor model can also be equipped with a human-computer interaction page. This page provides users with an intuitive and visual configuration interface, allowing users to easily configure and manage file processing processes to meet diverse file processing needs.
[0074] In summary, the file monitor model not only has powerful file monitoring and processing functions, but also provides users with a convenient configuration and management experience through the human-computer interaction page.
[0075] 2. File Processor Model
[0076] The file processor model, as the core of file information conversion in the system, has the core task of converting file information into a format that is standardized by the microservice ecosystem. After the conversion is completed, the information will be transmitted to the Kafka message queue for use by downstream consumer systems.
[0077] In this model, file mapping information is specially maintained. This mapping information is associated with the information of the file monitoring module, so that the system can accurately find and parse the field mapping relationship of each configured file. Based on these mapping relationships, the file processor model can convert file information into a standard message format.
[0078] In order to improve user experience and configuration efficiency, a human-computer interaction interface can also be provided, through which users can intuitively operate the mapping configuration of file fields, such as adding new mapping relationships, viewing the current configuration, or making necessary modifications. This flexible configuration method greatly enhances the applicability and ease of use of the file processor model.
[0079] 3. Consumption system model
[0080] The consumer system model is a service entity independent of the microservice cluster, and plays the role of a consumer of standardized data information in the Kafka queue in the embodiment of the present application. Based on its own configuration information, the model monitors specific topics in Kafka and obtains text information from them. Furthermore, the consumer system uses subtopics to accurately identify the obtained information.
[0081] After identification, the consumption system model performs subsequent processing or processing operations on the file information according to its own business needs. These operations may include but are not limited to data storage, analysis, conversion, etc. In the embodiment of the present application, a typical consumption method is to store the file information in a warehouse for subsequent data management and query.
[0082] Through this consumption system model, the embodiment of the present application realizes the effective consumption and utilization of standardized data information in the Kafka queue, meets the diverse needs of different microservices in data processing, and improves the flexibility and scalability of the entire system.
[0083] 4. Status Information Recording Model
[0084] The state information recording model is a recording component in the file processing life cycle, responsible for real-time tracking and recording the state changes of files. Once a file is monitored by the system, the model will immediately create corresponding information monitoring records. These records not only record the initial state of the file, but also continue to update as the file is processed in the system, such as transmission, conversion, consumption, etc.
[0085] When the consumption system model processes information, it will constantly interact with the status information recording model to ensure that the record is synchronized with the actual processing status. In this way, users can query the current processing status of the file at any time through the provided interactive interface, thereby achieving transparent monitoring of the file processing process.
[0086] 5. Normalized Information Model
[0087] The standardized information model defined in the embodiment of the present application refers to the information format stored in the Kafka message queue. This information is encoded in JSON format to ensure cross-platform and data exchange compatibility. The information header contains file monitoring information related to the Kafka queue, such as the source of the file, processing time, etc. In addition, it also contains subtopic file theme sub-information, which is used to identify and distinguish file data under different topics.
[0088] By normalizing the information model, the embodiment of the present application realizes the standardized and unified management of file information, improves the consistency and efficiency of data processing within the system, and also provides a structured and easy-to-process data source for subsequent data analysis and mining.
[0089] See also Figure 3 , the embodiment of the present application provides another file processing method, which specifically includes the following steps:
[0090] Another file processing method provided by an embodiment of the present application may include three steps: triggering a file processing method, file information processing, and consuming file information.
[0091] 1. Trigger file processing method
[0092] During system startup, the system listener is started and a corresponding listener is generated for each monitoring path in the configuration. The main steps are:
[0093] Step 1: Find the file configuration set. Read the file monitoring configuration from the file monitoring entity and generate a file monitoring configuration information set.
[0094] Step 2: Create an array of path information to be monitored. Based on the file monitoring configuration information set, the system extracts the monitoring path information and uses it as the key of the data set, thereby deduplicating the path information and constructing an array of path information to be monitored.
[0095] Step 3: Create a listener based on the path information.
[0096] Step 4: Start the listener. The system creates corresponding file listeners according to the path information and starts these file listeners immediately.
[0097] Step 5: Monitor file changes. After being started, the file monitor continuously monitors changes in data files under the target path.
[0098] Step 6: After all paths are monitored, the file monitor is started.
[0099] When creating a listener, the system will use the same or hierarchical paths to avoid wasting system resources by starting the listener for the same path. After the system starts the listener, it can monitor the changes of data files under the target path and trigger the file processing method based on the file change event.
[0100] During the system startup process, the system will automatically start and activate the monitoring paths pre-configured in the database. Once the system is started, it will use advanced file monitoring models to track and identify any changes to the model information in real time. Once the system detects a new file monitoring path, it will immediately trigger the file monitoring process and create a new listener to achieve real-time monitoring of the new path.
[0101] Data model monitoring adds monitoring functions to the database to monitor all operation events in the database. When the system detects that a data model is added or modified, the system will obtain the path information in the newly added record. Then, the system will check whether there is an enabled listener for the path. If the system determines that the path has not been set up for monitoring, the system will immediately add a file listener to monitor the path.
[0102] 2. File information processing
[0103] Step 1: Grab the lock. Given that the system is deployed in a cloud cluster environment, file creation, modification, or deletion events under the same path may trigger multiple processes to detect the event at the same time. To avoid problems caused by concurrent processing, the embodiment of the present application uses a Redis lock mechanism to ensure the uniqueness of event processing, thereby avoiding repeated processing of files.
[0104] Step 2: File integrity check and copy. Before further processing, the file will first undergo an integrity check to ensure data accuracy. Afterwards, the file will be copied to the specified location to prepare for subsequent operations.
[0105] Step 3: Match the database configuration record by file name. The file name of the file to be processed will be matched with the regular expression of the monitoring file under the corresponding path. This step is to confirm the configuration information of the file to be processed through reverse search.
[0106] Step 4: Delete the source file and release the Redis lock. The source file will be deleted after the copy is completed. This is to prevent other processes from processing the same file repeatedly. At the same time, the Redis lock will be released to ensure that the processing flow can be triggered again when the same file name appears again.
[0107] Step 5: File decompression. According to the configuration information, the system will use the corresponding decompression tool to decompress the compressed file. If the file is in text format, there is no need to decompress it, and the relevant configuration can be set to empty. This step ensures that subsequent processing can target the actual content of the file.
[0108] Step 6: File information reading and mapping. The file information will be read line by line and mapped according to the field mapping table. The mapped information will be configured into the target VO and stored in the list. When the number of read lines reaches the preset upper limit or the file is finished reading, the subsequent operation will be performed.
[0109] Step 7: Update the total amount of information recorded in Redis. After processing the file information, the system will update the total amount of information recorded in Redis to ensure data synchronization and accuracy.
[0110] Step 8: The processed file information will be sent to the Kafka message queue for consumption by other message subscribers in the system.
[0111] File information processing is the key process of this device, which aims to convert file content into a standardized data format for the system, thereby promoting the effective use of data by the system. This process mainly covers three core stages: information reading, information segmentation, and field mapping, to achieve smooth conversion from text format to VO, usually Entity or DTO.
[0112] In the information reading stage, the system adopts a line-by-line reading method to load the file content into the system cache line by line. <string>Each row of information is stored in the form of
[0113] The information cutting stage is the process of processing a line of file information into an information list. The data completes the LIST in this process. <string>To List <List <string>> conversion. The cutting process supports general and special processes. For files in general formats, the system directly calls the preset cutting method; for situations that require special processing, the coder needs to write the corresponding method in advance and configure the corresponding method name in the file processor model. The system obtains and applies the corresponding processing method through BeanFactory according to the method name in the model to complete the information cutting function. The default file format processed by the system is "message1|!message2|!message3", where "|!" is used as an information separator. In the general method, the system can convert information in this format into a list form, such as "{"message1","message2","message3"}".
[0114] In the field mapping stage, the system performs in-depth processing on the processed information according to the field mapping information in the file processor model. This step converts the information into data in the key-value pair format, and then converts it into the target VO through reverse transcription technology. The field mapping information records in detail the field name corresponding to each position in the file (such as position1 (the first position field) maps to "key1"). Based on this, the system converts the List information into the standard Json format, for example, converting "{"message1", "message2", "message3"}" into "{"key1":"message1", "key2":"message2", "key3":"message3"}", and then completes the conversion of Json information to VO through reverse transcription, that is, "{"key1":"message1", "key2":"message2", "key3":"message3"}", into a VO with the format of "{key1, key2, key3}".
[0115] After the above conversion process, the system can efficiently provide VO in a standardized format, providing strong support for the efficient processing of subsequent functional data of the system.
[0116] 3. Consumption file information
[0117] Once the message arrives at the Kafka message queue, other message subscribers in the system can consume it, thereby realizing further processing or consumption of the file information. Common consumption methods include warehousing data information and displaying file information.
[0118] It can be seen that the other file processing method provided by the embodiment of the present application has the following advantages:
[0119] 1. Shortened development cycle. By comparison, it can be seen that the file processing method provided in the embodiment of the present application realizes the reuse of multiple key codes, thereby making the implementation of functions easier and shortening the development cycle.
[0120] 2. Real-time file processing: Compared with the method of calling through scheduled tasks, the processing of files in the embodiment of the present application is more timely and repeated calling is easier.
[0121] 3. High cohesion and low coupling. Using stream processing, the encapsulation of each module in the stream processing has high cohesion characteristics, and using Kafka middleware reduces the coupling of each module in the information processing process.
[0122] 4. Clarify the processing flow. The file information processing flow adopts the stream processing method to ensure the sequence of the code processing process and the consistency of the processing method.
[0123] 5. Clear file storage information statistics. The page-based development process is not only convenient for R&D personnel, but also can provide intuitive page data for subsequent operation and maintenance personnel, making the merging of new functions more convenient.
[0124] 6. Convenient configuration-based production. In addition to providing template-based function development, the embodiment of the present application also implements one-click export and one-click import of configurations. Configuration-based management makes production more convenient without the need for complex data changes.
[0125] 7. The intuitive file statistics page display can also allow users to more intuitively check the daily processing execution status of each file.
[0126] See also Figure 4 The present application also provides a file processing device, the device comprising:
[0127] A determination unit 401 is used to determine a file to be processed;
[0128] An acquisition unit 402 is used to acquire target configuration information of the file to be processed from a database;
[0129] A conversion unit 403, configured to convert the to-be-processed file into target file data based on the target configuration information, wherein the format of the target file data is a standardized format;
[0130] The sending unit 404 is used to send the target file data to a message queue.
[0131] Optionally, the conversion unit 403 is specifically configured to:
[0132] Read the file data of the file to be processed line by line;
[0133] Performing information segmentation on the file data to obtain file data to be converted;
[0134] The to-be-converted file data is converted into the target file data based on the target field mapping information in the target configuration information.
[0135] Optionally, the conversion unit 403 is further configured to:
[0136] The to-be-processed file is decompressed based on the target decompression information in the target configuration information.
[0137] Optionally, the acquiring unit 402 is specifically configured to:
[0138] Add a distributed lock to the file to be processed;
[0139] The target configuration information is acquired from the database based on the identification information of the file to be processed.
[0140] Optionally, the determining unit 401 is specifically configured to:
[0141] Determine the file path to be monitored according to the file configuration set;
[0142] Based on the file listener, monitor whether there is a file change in the file path to be monitored;
[0143] If there is a file change in the file path to be monitored, the changed file is determined as a file to be processed.
[0144] Optionally, the message queue is a Kafka message queue.
[0145] It can be seen from this that the embodiment of the present application provides a file processing device that can obtain target configuration information pre-configured for the file to be processed from a database, and can quickly convert the file to be processed into target file data in a standardized format based on the target configuration information, and finally send the converted target file data to a message queue for further processing, thereby improving the processing efficiency of file processing.
[0146] The embodiment of the present application also provides a computer device, including: a memory, a processor, and a bus system;
[0147] Wherein, the memory is used to store programs;
[0148] The processor is used to execute the program in the memory to implement any one of the file processing methods described above;
[0149] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
[0150] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute any one of the file processing methods described above.
[0151] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0152] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / string> < / string> < / string> < / string>
Claims
1. A file processing method, characterized in that: The method comprises: Identify files to be processed; Acquire target configuration information of the file to be processed from a database; Converting the to-be-processed file into target file data based on the target configuration information, wherein the format of the target file data is a standardized format; The target file data is sent to a message queue.
2. The method according to claim 1, characterized in that The converting the to-be-processed file into target file data based on the target configuration information includes: Read the file data of the file to be processed line by line; Performing information segmentation on the file data to obtain file data to be converted; The to-be-converted file data is converted into the target file data based on the target field mapping information in the target configuration information.
3. The method according to claim 2, characterized in that Before reading the file data of the to-be-processed file line by line, the method further includes: The to-be-processed file is decompressed based on the target decompression information in the target configuration information.
4. The method according to claim 1, characterized in that: The step of obtaining target configuration information of the file to be processed from a database includes: Adding a distributed lock to the file to be processed; The target configuration information is acquired from the database based on the identification information of the file to be processed.
5. The method according to claim 1, characterized in that The step of determining the files to be processed includes: Determine the file path to be monitored according to the file configuration set; Based on the file listener, monitor whether there is a file change in the file path to be monitored; If there is a file change in the file path to be monitored, the changed file is determined as a file to be processed.
6. The method according to claim 1, characterized in that The message queue is a kafka message queue.
7. A file processing device, characterized in that: The device comprises: A determination unit, used for determining a file to be processed; An acquisition unit, used for acquiring target configuration information of the file to be processed from a database; A conversion unit, configured to convert the to-be-processed file into target file data based on the target configuration information, wherein the format of the target file data is a standardized format; The sending unit is used to send the target file data to the message queue.
8. The device according to claim 7, characterized in that The conversion unit is specifically used for: Read the file data of the file to be processed line by line; Performing information segmentation on the file data to obtain file data to be converted; The to-be-converted file data is converted into the target file data based on the target field mapping information in the target configuration information.
9. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory to implement the method according to any one of claims 1 to 6; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
10. A computer-readable storage medium, characterized in that: The device stores instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.