Multi-file data processing method and device, electronic equipment and storage medium
By extracting and importing data to be processed files and processing data with target shell scripts, the problem of frequent middleware updates caused by operating system version updates is solved, and the efficiency and maintainability of multi-file data processing is improved.
Patent Information
- Application Number
- CN202510202839.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-24
- Publication Date
- 2025-06-10
AI Technical Summary
In the prior art, due to the fast update speed of operating system versions, the deployed middleware needs to be updated frequently, which affects the coherent operation of multi-file processing processes and reduces data processing efficiency.
By extracting the acquired pending files, determining the pending data and its data type identification, and calling the pre-configured jar package to import the data into the SQLite middleware that comes with the Linux operating system, and executing SQL statements in the target shell script for data processing through SQLite drivers.
Database operations can be performed without independent server processes, which simplifies the data processing process, is easy to maintain and upgrade, ensures the coherent operation of multi-file processing, and improves data processing efficiency.
Smart Images
Figure CN120122987A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data processing, and particularly to a method, apparatus, electronic device, and storage medium for processing data of multiple files. Background Art
[0002] With the refinement of network operation and maintenance management, the metric data that the network management needs to monitor is increasing continuously. These data come from files of different manufacturer interfaces, and it is necessary to process the data in these files separately, cross-process, merge-process, or mix-process.
[0003] In the prior art, middleware and corresponding support software are usually deployed in the operating system to execute a multi-file processing program to process the data in multiple files. However, due to the improvement of the operating system version update speed, the deployed middleware also needs to be updated frequently, which affects the coherent operation of the multi-file processing process and reduces the data processing efficiency. Summary of the Invention
[0004] Based on the above problems, the present application provides a method, apparatus, electronic device, and storage medium for processing data of multiple files, aiming to improve the data processing efficiency.
[0005] The embodiments of the present application disclose the following technical solutions:
[0006] In a first aspect, a method for processing data of multiple files, the method includes:
[0007] Extract data from the obtained file to be processed, and determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed;
[0008] Call a pre-configured jar package, and import the data to be processed and the data type identifier corresponding to the data to be processed into a target data table in the SQLite middleware; the SQLite middleware is a built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute a target shell script;
[0009] Call the jar package and execute the SQL statement in the target shell script to process the target data in multiple target data tables to obtain a processing result; the target shell script is configured from a general shell script according to the user's file processing requirements.
[0010] Optionally, in the method as described above, the calling a pre-configured jar package and importing the data to be processed and the data type identifier corresponding to the data to be processed into a target data table in the SQLite middleware includes:
[0011] Call the pre-configured jar package, and based on the file identifier of the file to be processed, determine the data table parameters corresponding to the file identifier; the data table parameters are used to identify the data table corresponding to the file identifier to be processed.
[0012] Import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware.
[0013] Optionally, in the method as described above, the data extraction of the file to be processed obtained to determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed includes:
[0014] Obtain the data dimension, the header row, and the data rows in the file to be processed; the data rows include a sample data row and at least one data row to be processed; the data type identifiers in the header row are used to identify the data type corresponding to each data in any data row in sequence.
[0015] When it is determined that the data dimension is the same as the preset dimension, split the sample data row according to the preset delimiter to obtain multiple sample data.
[0016] When it is determined that the number of the sample data is the same as the number of the preset data types, perform split processing on the multiple data rows to be processed respectively to obtain the data to be processed and the data type identifier corresponding to the data to be processed.
[0017] Optionally, in the method as described above, the data dimension includes a time dimension, a space dimension, a geographical dimension, and a network element level dimension.
[0018] Optionally, in the method as described above, the method for obtaining the target shell script includes:
[0019] When it is determined that the data dimensions of multiple files to be processed are the same, use the obtained general shell script as the target shell script.
[0020] When it is determined that there are at least two data dimensions among the data dimensions of the multiple files to be processed, obtain the modified SQL statement.
[0021] Modify the general shell script based on the modified SQL statement to obtain the target shell script.
[0022] Optionally, in the method as described above, the SQL statement in the target shell script at least includes the identifier of the file to be processed, the target data, and the data type identifier of the target data.
[0023] Optionally, in the method as described above, the method further includes:
[0024] Call the export parameters in the jar package and export the processing result to a preset target file.
[0025] In a second aspect, a multi-file data processing device includes:
[0026] A data acquisition module, configured to perform data extraction on the acquired files to be processed, determine the data to be processed corresponding to the files to be processed and the data type identifier corresponding to the data to be processed;
[0027] A data storage module, configured to call a pre-configured jar package, import the data to be processed and the data type identifier corresponding to the data to be processed into a target data table in an SQLite middleware; the SQLite middleware is a built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute a target shell script;
[0028] A data processing module, configured to call the jar package and execute the SQL statements in the target shell script to perform data processing on the target data in multiple target data tables to obtain a processing result; the target shell script is configured from a general shell script according to the user's file processing requirements.
[0029] In a third aspect, the present application provides an electronic device, which includes: a processor and a memory communicatively connected to the processor;
[0030] The memory stores computer-executable instructions;
[0031] The processor executes the computer-executable instructions stored in the memory to implement the multi-file data processing method according to any one of the above embodiments.
[0032] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the multi-file data processing method according to any one of the above embodiments.
[0033] Compared with the prior art, the present application has the following beneficial effects:
[0034] The method of the present application extracts data from the obtained file to be processed, determines the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed; calls the pre-configured jar package, and imports the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the built-in SQLite middleware of the Linux operating system. The jar package is driven by the SQLite middleware to execute the target shell script, and there is no need for an independent server process, and database operations can be directly performed in the application program. Then call the jar package and execute the SQL statement in the target shell script to process the target data in multiple target data tables to obtain a processing result; the target shell script is configured from a general shell script according to the user's file processing requirements. When it is necessary to update the data processing logic or optimize the performance, the user only needs to update the jar package or modify the shell script, which is easy to maintain and upgrade, thereby ensuring the continuous operation of the multi-file processing process and improving the efficiency of data processing. Description of the Drawings
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0036] Figure 1 It is a schematic flowchart of an embodiment of a multi-file data processing method provided by the present application;
[0037] Figure 2 It is a schematic flowchart of another embodiment of a multi-file data processing method provided by the present application;
[0038] Figure 3 It is a schematic flowchart of still another embodiment of a multi-file data processing method provided by the present application;
[0039] Figure 4 It is a schematic structural diagram of an embodiment of a multi-file data processing device provided by the present application;
[0040] Figure 5 It is a schematic structural diagram of an embodiment of an electronic device provided by the present application. Detailed Embodiments
[0041] As described above, with the accelerating speed of operating system version updates, the middleware deployed also needs to be iteratively upgraded frequently. This change poses challenges to the coherent execution of the multi-file processing process, thereby leading to a decline in data processing efficiency.
[0042] After research, the inventors propose a multi-file data processing method, device, electronic device, and storage medium to solve the technical problem of low data processing efficiency.
[0043] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.
[0044] See Figure 1 , which is a schematic flowchart of an embodiment of a multi-file data processing method provided by this application. As Figure 1 shown, the method includes:
[0045] S101: Extract data from the obtained file to be processed, and determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed.
[0046] In this embodiment, after obtaining the data dimension, header row, and data row in the file to be processed, compare the data dimension in the file to be processed with the preset dimension to determine that the data dimension is consistent with the customer requirements. When it is determined that the data dimension is the same as the preset dimension, split the sample data row according to the preset delimiter to obtain multiple sample data. Compare the number of the split sample data with the number of the preset data types in the customer requirements. When it is determined that the number of the sample data is the same as the number of the preset data types, perform split processing on multiple data rows to be processed respectively to obtain the data to be processed and the data type identifier corresponding to the data to be processed.
[0047] S102: Call the pre-configured jar package, and import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware.
[0048] Among them, the SQLite middleware is the built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute the target shell script.
[0049] In this embodiment, a pre-configured jar package is called. Based on the file identifier of the file to be processed, the data table parameters corresponding to the file identifier are determined. Then, the data to be processed and the data type identifier corresponding to the data to be processed are imported into the target data table in the SQLite middleware.
[0050] S103: Call the jar package and execute the SQL statements in the target shell script to process the target data in multiple target data tables to obtain a processing result.
[0051] Among them, the target shell script is configured from a general shell script according to the user's file processing requirements.
[0052] In this embodiment, after obtaining the target shell script, the jar package is called to execute the SQL statements in the shell script. Then, the target data in multiple target data tables is processed according to the SQL statements to obtain a processing result.
[0053] In this embodiment, by extracting data from the obtained file to be processed, the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed are determined; a pre-configured jar package is called to import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the built-in SQLite middleware of the Linux operating system. The jar package is driven by the said SQLite middleware to execute the target shell script, and an independent server process is not required. Database operations can be directly performed in the application program. Then, the jar package is called again, and the SQL statements in the target shell script are executed to process the target data in multiple target data tables to obtain a processing result; the target shell script is configured from a general shell script according to the user's file processing requirements. When it is necessary to update the data processing logic or optimize the performance, the user only needs to update the jar package or modify the shell script, which is easy to maintain and upgrade. Furthermore, the coherent operation of the multi-file processing process is ensured, and the efficiency of data processing is improved.
[0054] See Figure 2 , which is a schematic flowchart of another embodiment of a multi-file data processing method provided by this application. As Figure 2 shown, different from Figure 1 , Figure 2 shows a specific implementation manner of "calling a pre-configured jar package and importing the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware" in S102, which is specifically introduced in combination with Figure 2 S1021 - S1022 therein.
[0055] S1021: Call the pre-configured jar package, and based on the file identifier of the file to be processed, determine the data table parameters corresponding to the file identifier.
[0056] Among them, the data table parameters are used to identify the data table corresponding to the file identifier to be processed.
[0057] In this embodiment, for example, 6 parameters are pre-configured in the pre-configured jar package to support the import of data in the file into SQLite. Among them, the parameter descriptions are shown in Table 1. By calling the pre-configured jar package and based on the file identifier of the file to be processed, the data table parameters corresponding to the file identifier are determined, that is, the SQLite target table in the table item.
[0058] Table 1 Parameter Description of the Jar Package
[0059]
[0060]
[0061] S1022: Import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware.
[0062] In this embodiment, establish a connection with the target table, then parse the data according to the data type identifier, and use SQL statements to insert the data to be processed and the data type identifier corresponding to the data to be processed into the table to complete the data import.
[0063] In this embodiment, by calling the pre-configured jar package and based on the file identifier of the file to be processed, the data table parameters corresponding to the file identifier are determined; then the data to be processed and the data type identifier corresponding to the data to be processed are imported into the target data table in the SQLite middleware, realizing an automated data processing process, reducing the steps of manual operation and human errors, reducing the time for data import and processing, and improving the overall work efficiency.
[0064] See Figure 3 , which is a schematic flowchart of another embodiment of a multi-file data processing method provided by this application. Different from Figure 1 , Figure 3 shows a specific implementation manner of "extracting data from the obtained file to be processed to determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed" in S101, which is specifically introduced in combination with Figure 3 S1011 - S1013 in.
[0065] S1011: Obtain the data dimension, header row, and data row in the file to be processed.
[0066] Among them, the data rows include a sample data row and at least one data row to be processed; the data type identifiers in the header row are used in sequence to identify the data type corresponding to each data in any data row.
[0067] The data dimensions include a time dimension, a space dimension, a geographical dimension, and a network element level dimension.
[0068] In this embodiment, for example, the resource metric file is a file related to the service gateway control plane metrics. Its corresponding space dimension is the parent network element granularity, the corresponding time dimension is 1 day, and the header row is BeginTime, EndTime, OMC_ID, Dn, UserLabel, MaxBearers; the sample data row in the data row contains 2024-10-30 06:01:29, 2024-10-30 06:01:29, 44017115, "DC=ManagedElement=969658_19,ServingGwcFunction=1", APP-HNGZgdS MF015BZX-15AZX012, 6550000. The meanings represented by different metrics in the header row are shown in Table 2.
[0069] Table 2 Metric Meaning Table
[0070]
[0071] S1012: When it is determined that the data dimension is the same as the preset dimension, the sample data row is split according to the preset delimiter to obtain multiple sample data.
[0072] In this embodiment, when the data dimension matches the preset dimension, the program reads the sample data row and splits it into multiple fields using the preset delimiter (such as a comma, a tab, etc.). Each field is a sample data.
[0073] S1013: When it is determined that the number of sample data is the same as the number of preset data types, the splitting process is respectively performed on multiple data rows to be processed to obtain the data to be processed and the data type identifiers corresponding to the data to be processed.
[0074] In this embodiment, after it is determined that the number of sample data is consistent with the number of preset data types, the program traverses the data rows to be processed and splits each row of data into multiple fields according to the preset delimiter to obtain the data to be processed and the data type identifiers corresponding to the data to be processed.
[0075] In this embodiment, by obtaining the data dimension, the header row, and the data rows in the file to be processed, when it is determined that the data dimension is the same as the preset dimension, the exemplary data rows are split according to the preset delimiter to obtain multiple pieces of exemplary data. When it is determined that the number of exemplary data is the same as the number of preset data types, the multiple data rows to be processed are respectively split to obtain the data to be processed and the data type identifiers corresponding to the data to be processed, realizing automatic splitting after data verification and consistency verification, reducing the steps of manual operations, and improving the processing efficiency.
[0076] Further, on the basis of the above embodiment, after S102 "invoke the pre-configured jar package and import the data to be processed and the data type identifiers corresponding to the data to be processed into the target data table in the SQLite middleware", and before S103 "invoke the jar package and execute the SQL statements in the target shell script to perform data processing on the target data in the multiple target data tables to obtain the processing result", the method may further include:
[0077] When it is determined that the data dimensions of the multiple files to be processed are the same, the obtained general shell script is used as the target shell script.
[0078] Among them, the SQL statements in the target shell script at least include the identifier of the file to be processed, the target data, and the data type identifier of the target data.
[0079] In this embodiment, when performing the data processing task, first check the data dimensions of the multiple files to be processed. If the data dimensions of the multiple files to be processed are the same, directly use the predefined general shell script as the target script, and this script contains standard SQL processing logic.
[0080] When it is determined that there are at least two data dimensions among the data dimensions of the multiple files to be processed, obtain the modified SQL statements.
[0081] In this embodiment, if there are at least two data dimensions among the data dimensions of the multiple files to be processed, obtain the SQL statements modified by the user himself. Subsequently, use these customized SQL statements to make necessary modifications to the general shell script, such as adjusting the SQL command part to adapt to the new data dimension.
[0082] Modify the general shell script based on the modified SQL statements to obtain the target shell script.
[0083] In this embodiment, for example, taking the calculation of a certain rate value = (molecule B / denominator B) * 100% as an example, the SQL statements prepared by the user obtained are as follows:
[0084] Select(table_A.Molecule B / table_B.Denominator B)*100% "Rate Value"
[0085] From table_A,table_B
[0086] Where table_A.Common A1 = table_B.Common A2
[0087] The SQL statement in the general shell script is as follows:
[0088] strSQL = "select xxx from table_A,table_B where table_A.A1 = table_B.B1"
[0089] Then the modified target shell script is as follows:
[0090] strSQL = "Select(table_A.Molecule B / table_B.Denominator B)*100% "Rate Value"
[0091] From table_A,table_B
[0092] Where table_A.Common A1 = table_B.Common A2"
[0093] In this embodiment, when it is determined that the data dimensions of multiple files to be processed are the same, the obtained general shell script is used as the target shell script; then, when it is determined that there are at least two data dimensions among the data dimensions of multiple files to be processed, the modified SQL statement is obtained, and the general shell script is modified based on the modified SQL statement to obtain the target shell script. When there are at least two different data dimensions, by obtaining the modified SQL statement and modifying the general shell script, various complex data processing requirements can be flexibly met, and the flexibility of data processing can be improved.
[0094] Further, on the basis of the above embodiment, after S103 "call the jar package and execute the SQL statement in the target shell script to perform data processing on the target data in multiple target data tables to obtain a processing result", the method may further include:
[0095] Call the export parameters in the jar package to export the processing result to a preset target file.
[0096] In this embodiment, the exported parameters configured in the jar package are called, and the export format and path are specified through a Java program to execute the data export logic. The processed data in the SQLite database is read and converted into a specified format, and then written into a preset target file to complete the export of the processing result.
[0097] In this embodiment, the processing result is exported to a preset target file by calling the export parameters in the jar package to ensure the accuracy and integrity of data export.
[0098] See Figure 4 , which is a schematic structural diagram of an embodiment of a multi-file data processing device provided by this application. As Figure 4 shown in the solid-line box, the device 40 includes a data acquisition module 41, a data storage module 42, and a data processing module 43.
[0099] Among them, the data acquisition module 41 is used to extract data from the acquired file to be processed, and determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed;
[0100] The data storage module 42 is used to call a pre-configured jar package to import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware; the SQLite middleware is a built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute the target shell script;
[0101] The data processing module 43 is used to call the jar package and execute the SQL statements in the target shell script to process the target data in multiple target data tables to obtain a processing result; the target shell script is configured from a general shell script according to the user's file processing requirements.
[0102] The multi-file data processing device provided by the embodiment of this application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so they will not be elaborated here.
[0103] Further, on the basis of the above embodiment, the data storage module 42 is specifically used to call a pre-configured jar package, determine the data table parameters corresponding to the file identifier based on the file identifier of the file to be processed; the data table parameters are used to identify the data table corresponding to the file identifier to be processed; and import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware.
[0104] The data processing device for multiple files provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments. The implementation principles and beneficial effects are similar, and will not be elaborated here.
[0105] Further, on the basis of the above embodiments, the data acquisition module 41 is specifically configured to acquire the data dimension, the header row, and the data rows in the file to be processed; the data rows include a demonstration data row and at least one data row to be processed; the data type identifiers in the header row are used to identify the data types corresponding to each data in any data row in sequence; when it is determined that the data dimension is the same as the preset dimension, the demonstration data row is split according to the preset delimiter to obtain multiple pieces of demonstration data; when it is determined that the number of pieces of demonstration data is the same as the number of preset data types, split processing is respectively performed on multiple data rows to be processed to obtain the data to be processed and the data type identifiers corresponding to the data to be processed.
[0106] The data processing device for multiple files provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments. The implementation principles and beneficial effects are similar, and will not be elaborated here.
[0107] Further, on the basis of the above embodiments, as Figure 4 shown by the dashed box in, the device 40 may further include a shell script acquisition module 44.
[0108] The shell script acquisition module 44 is configured to use the acquired general shell script as the target shell script when it is determined that the data dimensions of multiple files to be processed are the same; when it is determined that there are at least two data dimensions among the data dimensions of multiple files to be processed, acquire the modified SQL statement; and modify the general shell script based on the modified SQL statement to obtain the target shell script.
[0109] The data processing device for multiple files provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments. The implementation principles and beneficial effects are similar, and will not be elaborated here.
[0110] Further, on the basis of the above embodiments, the device 40 may further include an export module 45.
[0111] The export module 45 is configured to call the export parameters in the jar package and export the processing result to a preset target file.
[0112] The data processing device for multiple files provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments. The implementation principles and beneficial effects are similar, and will not be elaborated here.
[0113] See Figure 5, This figure is a schematic structural diagram of an embodiment of an electronic device provided by an embodiment of the present application. The electronic device 50 may include: a processor 51 and a memory 52.
[0114] Among them, the processor 51 is communicatively connected to the memory 52, and the memory 52 is used to store computer-executable instructions; the processor 51 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the computer-executable instructions stored in the memory 52.
[0115] Optionally, the memory 52 can be either independent or integrated with the processor 51. Optionally, when the memory 52 is a device independent of the processor 51, the electronic device 50 may further include: a bus for connecting the above-mentioned devices.
[0116] This electronic device is used to execute the technical solutions in any of the foregoing method embodiments, and its implementation principle and technical effects are similar, and will not be described in detail here.
[0117] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above method, and its implementation principle and technical effects are similar, and will not be described in detail here.
[0118] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components referred to as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.
[0119] The above is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for processing data of multiple files, characterized in that: include: Extracting data from the acquired file to be processed, determining the data to be processed corresponding to the file to be processed and a data type identifier corresponding to the data to be processed; Call the preconfigured jar package to import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware; The SQLite middleware is a built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute the target shell script; The jar package is called, and the SQL statement in the target shell script is executed to process the target data in multiple target data tables to obtain processing results; the target shell script is obtained by configuring a general shell script according to the user's file processing requirements.
2. The method according to claim 1, characterized in that The calling of the preconfigured jar package to import the data to be processed and the data type identifier corresponding to the data to be processed into the target data table in the SQLite middleware includes: Calling a preconfigured jar package, based on the file identifier of the file to be processed, determining a data table parameter corresponding to the file identifier; the data table parameter is used to identify the data table corresponding to the file identifier to be processed; The data to be processed and the data type identifier corresponding to the data to be processed are imported into the target data table in the SQLite middleware.
3. The method according to claim 1, characterized in that The step of extracting data from the acquired file to be processed and determining the data to be processed corresponding to the file to be processed and a data type identifier corresponding to the data to be processed includes: Obtaining data dimensions, header rows, and data rows in the to-be-processed file; the data rows include a demonstration data row and at least one to-be-processed data row; the data type identifier in the header row is used in turn to identify the data type corresponding to each data in any data row; When it is determined that the data dimension is the same as the preset dimension, splitting the demonstration data row according to a preset separator to obtain a plurality of demonstration data; When it is determined that the number of the demonstration data is the same as the number of the preset data types, the multiple rows of data to be processed are split and processed respectively to obtain the data to be processed and data type identifiers corresponding to the data to be processed.
4. The method according to claim 3, characterized in that The data dimensions include time dimension, space dimension, geographic dimension and network element level dimension.
5. The method according to claim 1, characterized in that The method for obtaining the target shell script includes: When it is determined that the data dimensions of the multiple files to be processed are the same, the obtained common shell script is used as the target shell script; When it is determined that there are at least two data dimensions among the data dimensions of the multiple files to be processed, obtaining a modified SQL statement; The general shell script is modified based on the modified SQL statement to obtain a target shell script.
6. The method according to claim 5, characterized in that The SQL statement in the target shell script at least includes an identifier of the file to be processed, target data, and a data type identifier of the target data.
7. The method according to claim 1, characterized in that The method further comprises: The export parameters in the jar package are called to export the processing results to a preset target file.
8. A data processing device for multiple files, characterized in that: include: A data acquisition module is used to extract data from the acquired file to be processed, and determine the data to be processed corresponding to the file to be processed and the data type identifier corresponding to the data to be processed; A data storage module is used to call a preconfigured jar package to import the data to be processed and the data type identifier corresponding to the data to be processed into a target data table in the SQLite middleware; The SQLite middleware is a built-in middleware of the Linux operating system; the jar package is driven by the SQLite middleware to execute the target shell script; The data processing module is used to call the jar package and execute the SQL statement in the target shell script to process the target data in multiple target data tables to obtain processing results; the target shell script is obtained by configuring the general shell script according to the user's file processing requirements.
9. An electronic device, characterized in that: The device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.