Data processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method, electronic equipment, a storage medium and a program product. The method comprises the following steps: receiving a file processing request sent by a client, wherein the file processing request comprises a to-be-processed file, equipment information of target storage equipment and file information of the to-be-processed file; determining file dump logic according to the equipment information; determining file analysis logic and file conversion logic according to the file information; processing the to-be-processed file according to the file analysis logic and the file conversion logic to obtain target data corresponding to the to-be-processed file; and storing the target data to the target storage device according to the file dump logic. According to the method, the file processing request is received through the unified data processing framework, the analysis, conversion and dump logic of the to-be-processed file is determined according to the file information and the equipment information, unified processing of different types of to-be-processed files is achieved, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, electronic equipment, storage medium and program product. Background Art

[0002] In the informatization construction of manufacturing enterprises, it is necessary to collect and process data from different equipment (such as production equipment, testing instruments, etc.) and integrate them into the manufacturing execution system (MES) or laboratory information management system (LIMS).

[0003] Enterprises usually adopt customized development solutions, relying on programming languages ​​and related frameworks, independently develop data processing logic according to the characteristics of each device and specific business needs, and transmit data to the corresponding information system.

[0004] However, in the above technical solution, since data processing logic needs to be developed and maintained separately for each type of equipment, the versatility of data processing is poor. Summary of the invention

[0005] The present application provides a data processing method, an electronic device, a storage medium and a program product to solve the problem of poor versatility in data processing.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0007] Receive a file processing request sent by a client, the file processing request including a file to be processed, device information of a target storage device, and file information of the file to be processed, the file information including a file type, file description information, file size, and corresponding service information;

[0008] Determine file dump logic according to the device information;

[0009] Determine the file parsing logic and file conversion logic according to the file information;

[0010] Processing the to-be-processed file according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file;

[0011] According to the file dump logic, the target data is stored in the target storage device.

[0012] In a possible implementation, determining the file parsing logic and the file conversion logic according to the file information includes:

[0013] Processing the file description information, the file size and the business information through a preset model to predict at least one data type included in the file to be processed, the data type including: text type, picture type, video type, voice type and table type;

[0014] Determining the file parsing logic according to the file type and the at least one data type;

[0015] The file conversion logic is determined according to the device information and the at least one data type.

[0016] In a possible implementation, determining the file parsing logic according to the file type and the at least one data type includes:

[0017] Determine a file opening rule according to the file type, and obtain a file parsing rule corresponding to each data type, wherein the file opening rule is used to indicate a method for opening the file to be processed;

[0018] Determining the file parsing logic includes the file opening rule and the file parsing rule corresponding to each data type.

[0019] In a possible implementation, determining the file conversion logic according to the device information and the at least one data type includes:

[0020] Determining a target data format supported by the target storage device according to the device information;

[0021] For any data type, according to the data type and the target data format, determine a file conversion rule corresponding to the data type, wherein the conversion rule is used to convert the data corresponding to the data type into the target data format;

[0022] Determining the file conversion logic includes file conversion rules corresponding to each data type.

[0023] In a possible implementation, the to-be-processed file is processed according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file, including:

[0024] Parsing the to-be-processed file according to the file parsing logic to obtain first parsed data;

[0025] Acquire first user correction information corresponding to the first parsed data, and update the first parsed data according to the first user correction information to obtain second parsed data;

[0026] Performing file conversion processing on the second parsed data by the file conversion logic to obtain first converted data;

[0027] Second user correction information corresponding to the first conversion data is acquired, and the first conversion data is updated according to the second user correction information to obtain the target data.

[0028] In a possible implementation manner, updating the first parsed data according to the first user correction information to obtain second parsed data includes:

[0029] Performing semantic analysis on the first user correction information to obtain semantic information of the first user correction information;

[0030] Determine, in the first parsed data, at least one piece of data to be modified and replacement data corresponding to each piece of data to be modified according to the semantic information;

[0031] In the first parsed data, each piece of data to be modified is updated to corresponding replacement data to obtain the second parsed data.

[0032] In a possible implementation, storing the target data to the target storage device according to the file dump logic includes:

[0033] Determine the storage interface corresponding to the target storage data according to the file dump logic;

[0034] The storage interface is called to store the target data in the target storage device.

[0035] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0036] A receiving module, configured to receive a file processing request sent by a client, wherein the file processing request includes a file to be processed, device information of a target storage device, and file information of the file to be processed, wherein the file information includes a file type, file description information, file size, and corresponding service information;

[0037] A first determining module, used to determine a file dump logic according to the device information;

[0038] A second determination module, used to determine the file parsing logic and the file conversion logic according to the file information;

[0039] A processing module, used to process the to-be-processed file according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file;

[0040] A storage module is used to store the target data to the target storage device according to the file dump logic.

[0041] In a possible implementation manner, the second determining module is specifically configured to:

[0042] Processing the file description information, the file size and the business information through a preset model to predict at least one data type included in the file to be processed, the data type including: text type, picture type, video type, voice type and table type;

[0043] Determining the file parsing logic according to the file type and the at least one data type;

[0044] The file conversion logic is determined according to the device information and the at least one data type.

[0045] In a possible implementation manner, the second determining module is specifically configured to:

[0046] Determine a file opening rule according to the file type, and obtain a file parsing rule corresponding to each data type, wherein the file opening rule is used to indicate a method for opening the file to be processed;

[0047] Determining the file parsing logic includes the file opening rule and the file parsing rule corresponding to each data type.

[0048] In a possible implementation manner, the second determining module is specifically configured to:

[0049] Determining a target data format supported by the target storage device according to the device information;

[0050] For any data type, according to the data type and the target data format, determine a file conversion rule corresponding to the data type, wherein the conversion rule is used to convert the data corresponding to the data type into the target data format;

[0051] Determining the file conversion logic includes file conversion rules corresponding to each data type.

[0052] In a possible implementation manner, the processing module is specifically configured to:

[0053] Parsing the to-be-processed file according to the file parsing logic to obtain first parsed data;

[0054] Acquire first user correction information corresponding to the first parsed data, and update the first parsed data according to the first user correction information to obtain second parsed data;

[0055] Performing file conversion processing on the second parsed data by the file conversion logic to obtain first converted data;

[0056] Second user correction information corresponding to the first conversion data is acquired, and the first conversion data is updated according to the second user correction information to obtain the target data.

[0057] In a possible implementation manner, the processing module is specifically configured to:

[0058] Performing semantic analysis on the first user correction information to obtain semantic information of the first user correction information;

[0059] Determine, in the first parsed data, at least one piece of data to be modified and replacement data corresponding to each piece of data to be modified according to the semantic information;

[0060] In the first parsed data, each piece of data to be modified is updated to corresponding replacement data to obtain the second parsed data.

[0061] In a possible implementation manner, the storage module is specifically used to:

[0062] Determine the storage interface corresponding to the target storage data according to the file dump logic;

[0063] The storage interface is called to store the target data in the target storage device.

[0064] In a third aspect, an embodiment of the present application provides a data processing device, including: a memory, a processor;

[0065] The memory stores computer-executable instructions;

[0066] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0067] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0068] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0069] The data processing method, electronic device, storage medium and program product provided in the embodiments of the present application are intended to improve the processing capabilities of different types of files through a unified data processing framework, and to extract the file to be processed and the file information corresponding to the file to be processed and the device information of the target device by receiving the file processing request sent by the client, and based on the file information and the device information, the corresponding file parsing logic, file conversion logic and file dump logic can be determined to ensure that the corresponding data processing can be performed according to the characteristics of the specific file, which is conducive to improving the efficiency of data processing. In addition, the target data after parsing and conversion can be accurately stored in the specified target storage device according to the preset dump logic, which is also conducive to ensuring the integrity and consistency of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0071] Figure 1 A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application;

[0072] Figure 2 Schematic diagram of the data processing method provided in the embodiment of the present application Figure 1 ;

[0073] Figure 3 A schematic diagram of a process for determining file solution logic and file conversion logic provided in an embodiment of the present application;

[0074] Figure 4 Schematic diagram of the data processing method provided in the embodiment of the present application Figure 2 ;

[0075] Figure 5 A schematic diagram of a graphical interface provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a data processing flow framework provided for an embodiment of the present application;

[0077] Figure 7 A schematic diagram of the structure of a data processing device 10 provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application.

[0079] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0081] In the technical solution of this application, the data collection, analysis, storage and other processing involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0082] In this application, "exemplarily", "in this embodiment", etc. are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0083] The descriptions such as "first" and "second" in this application are only used to illustrate and distinguish the objects described. There is no order, nor does it indicate a special limitation on the number of objects in the embodiments of this application, and it cannot constitute any limitation on the embodiments of this application. For example, the descriptions such as first parsed data and second parsed data are only used to distinguish different parsed data, and do not indicate different degrees of importance of the parsed data.

[0084] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0085] In order to clearly understand the technical solution of this application, Figure 1 , describe the application scenarios to which this application is applicable.

[0086] Figure 1 This is a schematic diagram of an application scenario of the data processing method provided in the embodiment of the present application. Figure 1, including a requesting device 101 and a data processing device 102. The requesting device 101 may be a local terminal device, and a file processing request may be sent to the data processing device 102 through the requesting device 101. The data processing device 102 may be a remote terminal device, and the data processing device 102 may perform data processing according to the file processing request sent by the requesting device 101, and store the target data obtained after the data processing in the target storage device. Optionally, the data processing device 102 may respond to the requesting device with a data processing status.

[0087] In related technologies, when it is necessary to collect and dump data from different types of devices, a customized development solution is usually adopted, that is, the data processing logic is independently developed using programming languages ​​and frameworks according to the characteristics of each device and specific business needs. In the above process, due to the lack of a unified data processing process, each type of device needs to develop and maintain its own data processing logic separately. This not only increases the complexity of data processing, but also leads to code redundancy and increased maintenance costs. In addition, when business needs or device types change, the data processing logic of the corresponding device needs to be redeveloped or modified. This approach that relies on specific device types will also limit the flexibility and efficiency of data processing.

[0088] In view of the above problems, in an embodiment of the present application, in order to solve the problem of poor versatility in data processing, it is possible to process different types of files through a unified interface and a standardized processing flow by receiving a file processing request sent by a client and extracting detailed information of the file to be processed (such as file type, description information, size, and business-related information). In addition, a unified data processing flow can not only improve the efficiency of data processing, but also reduce development costs. In order to improve the flexibility of data processing, the parsing logic and conversion logic that match the file characteristics can be determined based on the extracted file information, and the dump logic can be determined based on the device information of the target storage device, so that the data processing process is easy to expand and adjust.

[0089] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0090] Figure 2 The process of the data processing method provided in the embodiment of the present application Figure 1 . See Figure 2 , the method may include the following steps:

[0091] S201. Receive a file processing request sent by a client, where the file processing request includes a file to be processed, device information of a target storage device, and file information of the file to be processed.

[0092] The file information may include file type, file description information, file size, and corresponding service information.

[0093] The file type can be the format of the file, such as text files, binary files, image files, and video files. Different types of files require different data processing logic. The file description information and the business information corresponding to the file can be the business background and purpose of the file. This information provides context for the role and function of the file in a specific application scenario, which helps to clarify the purpose of the file to be processed in a specific business process. The file size can be the disk space occupied by the file to be processed.

[0094] The client can be a device that initiates a file processing request. For example, a data collection software or application can be deployed in a host computer (usually a computer or server), and the host computer can be regarded as a client.

[0095] The target storage device may be a device for storing processed data, and accordingly, the device information may be characteristic information of the target storage device, such as device type, device identifier, storage path, supported data formats, and access permissions.

[0096] Specifically, the client can read the preset data collection configuration file, obtain the file path, file type, collection method and other information of the files to be collected, and perform data collection in the local file system or other data source based on this information, obtain the files to be processed, and package these files to be processed and the file information corresponding to the files to be processed and the device information of the target storage device into a file processing request, and send it to the data processing device.

[0097] S202: Determine file dump logic according to device information.

[0098] The file dump logic can be the rules and methods for storing the converted data in the target device or system. For example, the specific path and storage format of the dump can be specified, and the error handling mechanism can be set to record the problems encountered during the dump process.

[0099] S203: Determine the file parsing logic and the file conversion logic according to the file information.

[0100] The file parsing logic may be rules and methods for reading and understanding the contents of a file. For example, the file parsing logic may be determined based on the type of the file to be processed.

[0101] The file conversion logic can be the rules and methods for converting the parsed data into a specific format or structure so that the data can meet the requirements of subsequent processing and storage. For example, the parsed data can be cleaned to remove null values, duplicate values, or illegal data. Necessary conversion operations can also be applied to convert and calculate the data format.

[0102] The specific execution process of determining the file parsing logic and the file conversion logic in S203 can also be found in the present application Figure 3 The embodiment 2 shown is not described in detail here.

[0103] S204: Process the file to be processed according to the file parsing logic and the file conversion logic to obtain target data corresponding to the file to be processed.

[0104] The determined file parsing logic and file conversion logic are applied to the files to be processed, and the data reading, parsing, conversion and other operations are actually performed to generate the target data for the file dump process.

[0105] The specific execution process of S204 can also refer to S403 to S406 in Embodiment 3 of the present application, which will not be described in detail here.

[0106] S205: Store the target data into the target storage device according to the file dump logic.

[0107] As an optional implementation, the target data is stored in the target storage device through the following steps: determining the storage interface corresponding to the target storage data according to the file dump logic; and calling the storage interface to store the target data in the target storage device.

[0108] The storage interface can be an API, library, or command line tool that communicates with the target storage device. The storage interface defines how to upload, write, or store data to the target device. The corresponding data upload, write, or storage operations can be performed through the storage interface.

[0109] For example, the file dump logic may indicate that the target data is stored in the database of the target storage device, and the storage interface that communicates with the database of this type may be called to insert the target data in batches into the specified table of the database by executing the corresponding insert statement. For another example, the data may be forwarded to an external platform using a representational state transfer (REST) ​​interface (RESTful), and a request containing the target data is sent to a specified API endpoint by calling the corresponding hypertext transfer protocol (HTTP) request library. By dumping the target data according to the file dump logic, the corresponding storage interface may be selected and called according to different storage and forwarding requirements, ensuring that the data can be accurately and efficiently stored or forwarded to various target devices.

[0110] The data processing method provided by the embodiment of the present application receives a file processing request sent by a client, wherein the file processing request includes the file to be processed, the device information of the target storage device, and the file information of the file to be processed; determines the file dump logic according to the device information; determines the file parsing logic and the file conversion logic according to the file information; processes the file to be processed according to the file parsing logic and the file conversion logic to obtain the target data corresponding to the file to be processed; and stores the target data in the target storage device according to the file dump logic. By receiving the file processing request sent by the client and extracting the file to be processed, the file information, and the device information of the target storage device from it, the specific processing flow of each file to be processed can be determined based on this information, and unified processing of various types of files to be processed can be achieved, thereby improving the versatility of data processing and also being conducive to improving the efficiency of data processing.

[0111] Based on the above embodiments, Figure 3 The illustrated embodiment provides a detailed description of the process of determining data processing logic based on file information.

[0112] Figure 3 A schematic diagram of the process of determining the file parsing logic and the file conversion logic provided in the embodiment of the present application. Figure 3 , the method may include:

[0113] S301. Process file description information, file size, and business information through a preset model to predict at least one data type included in the file to be processed.

[0114] The data types may include: text type, picture type, video type, voice type and table type;

[0115] The preset model can be a classification model based on machine learning or a rule engine.

[0116] Specifically, the preset model can determine the type of data that may be contained in the file based on the file description information, file size and business information. For example, if the file description information contains "production data", the preset model may predict that the file to be processed contains table-type data; if the file description information contains "image data", it may predict that the file contains picture-type data.

[0117] The preset model can output one or more possible data types contained in the file to be processed.

[0118] S302: Determine file parsing logic according to the file type and at least one data type.

[0119] As an optional implementation, the file parsing logic is determined by the following steps: determining the file opening rules according to the file type, and obtaining the file parsing rules corresponding to each data type, the file opening rules are used to indicate the method of opening the file to be processed; determining that the file parsing logic includes the file opening rules, and the file parsing rules corresponding to each data type.

[0120] The file opening rule may indicate a specific method or tool for opening or reading a file to be processed. Different file types require different opening rules to ensure that the file can be read correctly.

[0121] For example, for a file to be processed whose file type is JSON, the file opening rule may indicate that "a JavaScript object notation (JSON) parser is required to read the file content as a dictionary or similar structure"; for a portable document format (PDF) file, the file opening rule may indicate that "a PDF reader is required to read text and image content".

[0122] The file parsing rules refer to the specific methods used to convert the contents of the file to be processed into a processable data structure.

[0123] For example, for text-type data, you can use a text parser to extract text content and perform word segmentation, keyword extraction, and other processing; for picture-type data, you can use an image parser to extract image metadata (such as resolution, color mode, etc.).

[0124] The file parsing logic may be a complete process for opening and parsing a file, including the file opening rules mentioned above, and file parsing rules corresponding to each data type in the file to be processed.

[0125] For example, for a CSV file, its file opening rule is determined to be "read the file content line by line using a CSV parser", and the file contains table-type data and text-type data. The file parsing rule corresponding to the table type can be "parse each line of CSV data into a dictionary object, with the key being the column name and the value being the corresponding cell content", and the file parsing rule corresponding to the text type can be "remove extra spaces and extract keywords". Accordingly, the file parsing logic of the CSV file can include the file opening rule and the file parsing rules corresponding to each data type.

[0126] S303: Determine file conversion logic according to device information and at least one data type.

[0127] As an optional implementation, the file conversion rules are determined by the following steps: determining the target data format supported by the target storage device based on the device information; for any data type, determining the file conversion rules corresponding to the data type based on the data type and the target data format, the conversion rules are used to convert the data corresponding to the data type into the target data format; determining the file conversion logic includes the file conversion rules corresponding to each data type.

[0128] The target data format may be a data format supported by the target storage device, for example, a structured query language (SQL) table format, an extensible markup language XML format, a binary file format, etc.

[0129] The file conversion rule may be a specific method for converting the data in the to-be-processed file from a current format into a target data format.

[0130] For example, for table data, if the target data format is a SQL table, the table data can be inserted into a table in a relational database; for text data, if the target format is JSON, the text content can be embedded into a field in a JSON object.

[0131] The file conversion logic may be a complete process of converting the data in the to-be-processed file into a target format, and may include conversion rules corresponding to each data type.

[0132] For example, for a CSV file containing table data and text data, based on the device information, it can be determined that the target storage device is a relational database and the target data format is a SQL table. Each row of data in the CSV file can be inserted into table A of the relational database, and the text content in the CSV file can be stored in column B of table A. The file conversion logic of the file can include file conversion rules corresponding to these two data types.

[0133] The process of determining the file processing logic provided by the embodiment of the present application processes the file description information, file size and business information through a preset model, predicts at least one data type included in the file to be processed, and the data types include: text type, picture type, video type, voice type and table type; determines the file parsing logic according to the file type and at least one data type; determines the file conversion logic according to the device information and at least one data type. By combining the file description information, file size and business information, the preset model can more accurately judge the content of the file, reduce the possibility of misjudgment, and help improve the accuracy of data processing. According to the file type and the predicted data type, the file opening rules and the parsing rules for each data type are determined to avoid data loss or damage caused by format mismatch or parsing errors. Using specific conversion rules for each data type ensures that the data will not be lost or deformed during the conversion process, ensuring the integrity and consistency of the data.

[0134] Based on the above embodiments, Figure 4 The embodiments shown further illustrate the technical solution of the present application.

[0135] Figure 4 Schematic diagram of the data processing method provided in the embodiment of the present application Figure 2 , see Figure 4 , the method specifically comprises:

[0136] S401: Receive a file processing request sent by a client, where the file processing request includes a file to be processed, device information of a target storage device, and file information of the file to be processed.

[0137] File information may include file type, file description information, file size, and corresponding service information.

[0138] The specific execution process of S401 can refer to S201 in Example 1, which will not be repeated here.

[0139] S402: Determine the file parsing logic and the file conversion logic according to the file information, and determine the file dumping logic according to the device information.

[0140] Optionally, in the process of determining the file parsing logic, file conversion logic and file dumping logic according to the file information and device information, the configuration information may also be referred to, that is, the file parsing logic, file conversion logic and file dumping logic may be determined according to the file information, device information and configuration information.

[0141] If the configuration information exists in the preset storage space, the configuration information can be obtained in the preset storage space. If the configuration information does not exist in the preset storage space, the configuration information can be generated through a graphical interface. For example, a graphical interface can be displayed, and in response to the user's input operation in the graphical interface, the configuration information input by the user is obtained.

[0142] A graphical user interface allows users to interact with a computer system through graphical elements such as windows, icons, and buttons. Compared to a command line interface, a graphical user interface can provide a more intuitive and user-friendly operating experience. Users can execute commands or configure settings by clicking and dragging the mouse without having to remember complex command syntax.

[0143] Figure 5 A schematic diagram of a graphical interface provided in an embodiment of the present application.

[0144] like Figure 5 As shown in the figure, the graphical interface can include a series of predefined functional components, such as file parsing logic input controls, file conversion logic input controls, and file dump logic input controls. Users can customize the specific steps of file parsing, file conversion, and file dumping by selecting or dragging according to specific business needs, ensuring that the logic of data processing can be flexibly adjusted.

[0145] It should be noted that in order to further simplify the configuration work, the embodiment of the present application may also include a preset template library. The preset template library contains configuration templates for different file types. Specifically, through the "Get Preset Template" component, a suitable template can be selected from the preset template library as a starting point according to specific needs. The selected template can be automatically parsed into the graphical interface. The user can make personalized adjustments based on the template according to the detailed parameter prompts of each component provided by the graphical interface. The graphical interface can also provide users with instant visual feedback to help users confirm whether their operations are correct. For example, after the user configures a step, the interface can display specific test examples so that the user can immediately see the effect of the configuration.

[0146] In response to the input operation of the "Submit Configuration Information" component, the user's configuration information may be acquired.

[0147] Among them, the execution steps of determining the file parsing logic and the file conversion logic according to the file information can refer to Example 2.

[0148] In this embodiment, the flexibility and efficiency of data processing can be further improved on the basis of Embodiment 2. By comprehensively considering file information (such as file type, file size, file description information and metadata, etc.) and user configuration information (such as rule settings, preference options, etc.), this combination makes the data processing logic more flexible, not only taking into account the characteristics of the file to be processed itself, but also fully responding to the personalized needs of users.

[0149] Specifically, the configuration information may be verified, and the verification status includes a pass status and a fail status. Verifying the configuration information may ensure the integrity and accuracy of the configuration information.

[0150] Optionally, the verification status includes integrity verification, consistency verification, legality verification and performance verification, wherein the integrity verification, consistency verification and legality verification must all pass, while the performance verification is optional and will not interfere with subsequent data processing even if it fails.

[0151] The purpose of integrity check is to ensure that the configuration information contains all the required parameters and that each required item has a valid value and cannot be omitted or left blank. For example, if a database is selected as the target storage device, you can check whether the configuration information provides the necessary connection information, such as user name, password, host address, port number, and database name.

[0152] The purpose of consistency checking is to ensure that there are no conflicts or incompatibilities between configuration items and to ensure the consistency of the entire configuration logic. For example, you can check whether the dependencies between configuration items are reasonable. Some components may require the output of the previous component as input, and you can verify whether this dependency is established.

[0153] The purpose of the validity check is to ensure that the values ​​in the configuration information conform to specific formats and rules, and to prevent the data processing flow from failing to complete normally due to format errors or illegal input. For example, you can check whether the uniform resource locator (URL) of the API conforms to the standard format, or check whether the format of the database connection character is correct.

[0154] The purpose of performance verification is to ensure that certain configuration items do not lead to excessive resource consumption or performance bottlenecks. For example, if configuration related to network transmission is involved (such as API calls, cloud storage uploads, etc.), the requirements for network bandwidth can be evaluated to ensure that the efficiency of data dumping is not affected by network bottlenecks.

[0155] If the verification status of the configuration information is passed, the file processing logic set by the user in the current configuration information can be used to replace or supplement the file processing logic determined according to the file information. If the verification status of the configuration information is not passed, a prompt message can be provided to the user in the graphical interface to prompt the user to modify the currently input configuration information. The graphical interface in this embodiment also supports real-time testing and debugging functions to ensure the efficiency and accuracy of the configuration.

[0156] In addition, in order to facilitate the reuse of configuration information and improve the efficiency of data processing, in an embodiment of the present application, the configuration information with a verification status of passed can be saved to a preset template library through the input operation of the "Save Configuration Information" component to facilitate the reuse of the configuration information.

[0157] S403: parse the file to be processed according to the file parsing logic to obtain first parsed data.

[0158] The first parsing data refers to the preliminary file parsing results.

[0159] For example, the current file A to be processed may be a CSV file. According to the file parsing logic, a CSV parser may be used to read the file A to be processed, and parse each row of data in A into a dictionary, where the key of the dictionary is the column name and the value is the corresponding field content. Accordingly, the generated first parsed data may be the data in the dictionary form corresponding to the data in A.

[0160] S404: Acquire first user correction information corresponding to the first parsed data, and obtain second parsed data according to the first user correction information.

[0161] As an optional implementation, the second parsed data is obtained by the following steps: performing semantic analysis on the first user correction information to obtain semantic information of the first user correction information; determining at least one piece of data to be modified and replacement data corresponding to each piece of data to be modified in the first parsed data based on the semantic information; in the first parsed data, updating each piece of data to be modified to the corresponding replacement data to obtain the second parsed data.

[0162] The first user correction information refers to an operation record that can be used to modify the first parsed data for the user, including error correction and supplementary information, etc. The first user correction information can be a natural language description, annotation, etc. For example, the first parsed data can be displayed on a graphical interface for the user to view, and the user can input the first user correction information in the graphical interface.

[0163] The semantic information may be specific modification intention and operation details extracted from the first user correction information input by the user, and may include the location of the data to be modified, the modification type, and the target value, etc.

[0164] The data to be modified may be a specific field or record in the first parsed data that needs to be modified.

[0165] The replacement data may be a new value specified by a user, and is used to replace the data to be modified in the first parsed data.

[0166] The second parsed data may be an improved and updated version of the first parsed data. It is understandable that if the first user correction information does not modify the first parsed data, the content of the second parsed data may be consistent with the first parsed data.

[0167] For example, the first parsed data may include field A, field B, and field C. The data corresponding to field A is a1, the data corresponding to field B is b1, and the data corresponding to field C is c1. A natural language processing model may be used to perform semantic analysis on the first user correction information input by the user to extract semantic information. For example, based on the semantic information, it may be determined that the data to be processed is field A and field B, the replacement data corresponding to field A is a2, and the replacement data corresponding to field B is b2. Therefore, the first parsed data is updated based on the semantic information, and in the obtained second parsed data, the data corresponding to field A is a2, the data corresponding to field B is b2, and the data corresponding to field C is c1.

[0168] S405 , performing file conversion processing on the second parsed data through file conversion logic to obtain first converted data.

[0169] Specifically, the second parsed data may be converted and processed using the file conversion rules indicated by the file conversion logic. For example, new calculated fields or auxiliary fields may be added according to business requirements. For example, the total amount may be calculated in the second parsed data of an order table and inserted into the second parsed data as a new field to form the first converted data.

[0170] S406: Acquire second user correction information corresponding to the first conversion data, and update the first conversion data according to the second user correction information to obtain target data.

[0171] The second user correction information may be an operation record of the user modifying the first conversion data.

[0172] Specifically, the first conversion data can be displayed to the user, allowing the user to modify and supplement the first conversion data according to specific needs. For example, the user wants to modify the value of a field in the first conversion data and add a new field. This modification and supplement operation is the second user correction information. According to the second user correction information, the first conversion data is updated to generate the target data.

[0173] S407: Store the target data in the target storage device according to the file dump logic.

[0174] The specific execution process of S407 can refer to S205 and will not be repeated here.

[0175] Figure 6 The data processing flow diagram provided in the embodiment of the present application is shown in FIG. Figure 6 The data processing device can receive at least one file processing request, and parse and process the to-be-processed file contained in the file processing request through the file parsing module and the file conversion module in sequence to obtain at least one target data. And the file dump module stores or forwards each target data to the target storage device.

[0176] in, Figure 6 The specific execution process involved can be found in the above-mentioned embodiments and will not be described again here.

[0177] Figure 7 This is a schematic diagram of the structure of the data processing device 10 provided in the embodiment of the present application. Figure 7 The data query device 10 includes: a receiving module 11, a first determining module 12, a second determining module 13, a processing module 14 and a storage module 15.

[0178] The receiving module 11 is used to receive a file processing request sent by a client.

[0179] The file processing request includes the file to be processed, the device information of the target storage device and the file information of the file to be processed. The file information includes the file type, file description information, file size and corresponding business information.

[0180] The first determination module 12 is used to determine the file dump logic according to the device information.

[0181] The second determination module 13 is used to determine the file parsing logic and the file conversion logic according to the file information.

[0182] The processing module 14 is used to process the file to be processed according to the file parsing logic and the file conversion logic to obtain the target data corresponding to the file to be processed.

[0183] The storage module 15 is used to store the target data into the target storage device according to the file dump logic.

[0184] In one implementation, the second determining module 13 is specifically configured to:

[0185] The file description information, file size and business information are processed through a preset model to predict at least one data type included in the file to be processed; the file parsing logic is determined based on the file type and at least one data type; and the file conversion logic is determined based on the device information and at least one data type.

[0186] The data types include: text type, picture type, video type, voice type and table type.

[0187] In one implementation, the second determining module 13 is specifically configured to:

[0188] According to the file type, the file opening rule is determined, and the file parsing rule corresponding to each data type is obtained; the file parsing logic is determined to include the file opening rule and the file parsing rule corresponding to each data type.

[0189] Among them, the file opening rule is used to open the file to be processed.

[0190] In one implementation, the second determining module 13 is specifically configured to:

[0191] Determine the target data format supported by the target storage device based on the device information; for any data type, determine the file conversion rule corresponding to the data type based on the data type and the target data format; determine the file conversion logic including the file conversion rule corresponding to each data type.

[0192] The conversion rule is used to convert data corresponding to the data type into a target data format.

[0193] In one embodiment, the processing module 14 is specifically configured to:

[0194] According to the file parsing logic, the file to be processed is parsed to obtain the first parsed data; the first user correction information corresponding to the first parsed data is obtained, and the first parsed data is updated according to the first user correction information to obtain the second parsed data; the second parsed data is converted according to the file conversion logic to obtain the first converted data; the second user correction information corresponding to the first converted data is obtained, and the first conversion data is updated according to the second user correction information to obtain the target data.

[0195] In one embodiment, the processing module 14 is specifically configured to:

[0196] Perform semantic analysis on the first user correction information to obtain semantic information of the first user correction information; determine at least one piece of data to be modified and replacement data corresponding to each piece of data to be modified in the first parsed data based on the semantic information; and update each piece of data to be modified in the first parsed data to the corresponding replacement data to obtain second parsed data.

[0197] In one embodiment, the storage module 15 is specifically used for:

[0198] According to the file dump logic, the storage interface corresponding to the target storage data is determined; and the storage interface is called to store the target data in the target storage device.

[0199] Figure 8 Schematic diagram of the structure of the electronic device 20 provided in the embodiment of the present application. Figure 8 As shown, the electronic device 20 may include:

[0200] Processor 21 and memory 22.

[0201] The processor 21 executes the computer-executable instructions stored in the memory 22, so that the processor 21 executes the solution in the above embodiment. For details, please refer to the relevant description in the above method embodiment, and no further details will be given here.

[0202] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0203] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0204] In this embodiment, the memory 22 is connected to the processor 21 via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0205] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0206] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the data processing method provided in any of the above embodiments. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0207] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the data processing method in the above embodiment.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0209] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0210] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0211] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0212] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0213] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0214] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Receive a file processing request sent by a client, the file processing request including a file to be processed, device information of a target storage device, and file information of the file to be processed, the file information including a file type, file description information, file size, and corresponding service information; Determine file dump logic according to the device information; Determine the file parsing logic and file conversion logic according to the file information; Processing the to-be-processed file according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file; According to the file dump logic, the target data is stored in the target storage device.

2. The method according to claim 1, characterized in that Determine the file parsing logic and file conversion logic according to the file information, including: Processing the file description information, the file size and the business information through a preset model to predict at least one data type included in the file to be processed, the data type including: text type, picture type, video type, voice type and table type; Determining the file parsing logic according to the file type and the at least one data type; The file conversion logic is determined according to the device information and the at least one data type.

3. The method according to claim 2, characterized in that Determining the file parsing logic according to the file type and the at least one data type includes: Determine a file opening rule according to the file type, and obtain a file parsing rule corresponding to each data type, wherein the file opening rule is used to indicate a method for opening the file to be processed; Determining the file parsing logic includes the file opening rule and the file parsing rule corresponding to each data type.

4. The method according to claim 2 or 3, characterized in that: Determining the file conversion logic according to the device information and the at least one data type includes: Determining a target data format supported by the target storage device according to the device information; For any data type, according to the data type and the target data format, determine a file conversion rule corresponding to the data type, wherein the conversion rule is used to convert the data corresponding to the data type into the target data format; Determining the file conversion logic includes file conversion rules corresponding to each data type.

5. The method according to any one of claims 1 to 4, characterized in that: Processing the to-be-processed file according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file includes: Parsing the to-be-processed file according to the file parsing logic to obtain first parsed data; Acquire first user correction information corresponding to the first parsed data, and update the first parsed data according to the first user correction information to obtain second parsed data; Performing file conversion processing on the second parsed data by the file conversion logic to obtain first converted data; Second user correction information corresponding to the first conversion data is acquired, and the first conversion data is updated according to the second user correction information to obtain the target data.

6. The method according to claim 5, characterized in that The first parsed data is updated according to the first user correction information to obtain second parsed data, including: Performing semantic analysis on the first user correction information to obtain semantic information of the first user correction information; Determine, in the first parsed data, at least one piece of data to be modified and replacement data corresponding to each piece of data to be modified according to the semantic information; In the first parsed data, each piece of data to be modified is updated to corresponding replacement data to obtain the second parsed data.

7. The method according to any one of claims 1 to 6, characterized in that: According to the file dump logic, storing the target data to the target storage device includes: Determine the storage interface corresponding to the target storage data according to the file dump logic; The storage interface is called to store the target data in the target storage device.

8. A data processing device, characterized in that: The device comprises: A receiving module, configured to receive a file processing request sent by a client, wherein the file processing request includes a file to be processed, device information of a target storage device, and file information of the file to be processed, wherein the file information includes a file type, file description information, file size, and corresponding service information; A first determining module, used to determine a file dump logic according to the device information; A second determination module, used to determine the file parsing logic and the file conversion logic according to the file information; A processing module, used to process the to-be-processed file according to the file parsing logic and the file conversion logic to obtain target data corresponding to the to-be-processed file; A storage module is used to store the target data to the target storage device according to the file dump logic.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images