Data file importing method and device, program product and electronic equipment

Abstract

The invention discloses a data file import method and device, a program product and electronic equipment, and relates to the field of financial science and technology or other related fields, and the import method comprises the steps of obtaining an application file of a target application, processing the application file to obtain a plurality of texts, creating a data job for each text, and creating a data job for each text; and asynchronously executing the data operation by adopting the distributed nodes, and importing the text into a preset data table in a preset database. According to the method and the device, the technical problem that the speed of importing the data file is relatively low in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and more specifically, to a method, apparatus, program product, and electronic device for importing data files. Background Technology

[0002] Every application needs data storage, and the larger the application, the more data it needs to store. Currently, many developers typically design different data import scripts for different data types, or they manually enter the data when it is relatively regular and not plentiful.

[0003] Then, during data collection, the same data may be stored in different formats. This situation and different data types require different import scripts to be written for adaptation, which will affect the progress of data entry, waste the time and energy of developers, and when this batch of data is urgently needed, it may not be entered in time, resulting in some unnecessary losses.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, program product, and electronic device for importing data files, to at least solve the technical problem of slow data file import speed in related technologies.

[0006] According to one aspect of the present invention, a method for importing data files is provided, comprising: obtaining an application file of a target application, wherein the application file records data generated during the operation of the target application; processing the application file to obtain multiple texts, wherein each text corresponds to a field; creating a data job for each text; and for each text, asynchronously executing the data job using distributed nodes to import the text into a preset data table in a preset database, wherein the preset data table is a data table constructed based on the field corresponding to the text.

[0007] Further, the steps of processing the application file to obtain multiple texts include: determining the file type of the application file, wherein the file type includes: image file type and text file type; if the application file is an image file type, scanning the application file to obtain scan data, and processing the scan data to obtain multiple texts; if the application file is a text file type, processing the application file to obtain multiple texts.

[0008] Further, the steps of processing the scanned data to obtain multiple texts include: extracting multiple fields from the scanned data and obtaining the data corresponding to each field from the scanned data; determining all field values ​​corresponding to the field based on the data corresponding to the field; and constructing the text corresponding to the field based on the field and all field values ​​corresponding to the field.

[0009] Further, the steps of processing the application file to obtain multiple texts include: extracting text data from the application file and recognizing the text data to obtain multiple fields; obtaining the data corresponding to each field from the text data and determining all field values ​​corresponding to the field based on the data corresponding to the field; splitting all field values ​​using preset symbols and constructing the text corresponding to the field based on the field and all the split field values.

[0010] Furthermore, before creating a data job for each text, the process includes: determining whether the text's data volume exceeds a preset data volume; if the text's data volume exceeds the preset data volume, splitting the text into multiple sub-texts, and representing each sub-text as a new text; and creating a data job for each new text and each text that does not exceed the preset data volume.

[0011] Furthermore, before using distributed nodes to asynchronously execute data jobs and import text into a preset data table in a preset database, the process includes: traversing the fields corresponding to each text and determining the field length of the currently traversed field, where the field length is the length of the longest field value among all field values ​​corresponding to the field; and based on the field and the field length, creating a data table associated with the field in the preset database.

[0012] Furthermore, the step of using distributed nodes to asynchronously execute data jobs and import text into a preset data table in a preset database includes: querying a matching preset data table from the preset database based on the fields corresponding to the data job; and using distributed nodes to import the text indicated by the data job into the preset data table.

[0013] According to another aspect of the present invention, a data file import apparatus is also provided, comprising: an acquisition unit for acquiring an application file of a target application, wherein the application file records data generated during the operation of the target application; a processing unit for processing the application file to obtain multiple texts, wherein each text corresponds to a field; a creation unit for creating a data job for each text; and an execution unit for asynchronously executing the data job using distributed nodes for each text, importing the text into a preset data table in a preset database, wherein the preset data table is a data table constructed based on the field corresponding to the text.

[0014] Furthermore, the processing unit includes: a first determining module, used to determine the file type of the application file, wherein the file type includes: image file type and text file type; a first processing module, used to scan the application file when the file type of the application file is an image file type, obtain scan data, and process the scan data to obtain multiple texts; and a second processing module, used to process the application file when the file type of the application file is a text file type to obtain multiple texts.

[0015] Furthermore, the first processing module includes: a first extraction submodule, used to extract multiple fields from the scanned data and obtain the data corresponding to each field from the scanned data; a first determination submodule, used to determine all field values ​​corresponding to the field based on the data corresponding to the field; and a first construction submodule, used to construct the text corresponding to the field based on the field and all field values ​​corresponding to the field.

[0016] Furthermore, the second processing module includes: a second extraction submodule, used to extract text data from the application file and recognize the text data to obtain multiple fields; a second determination submodule, used to obtain the data corresponding to each field from the text data and determine all field values ​​corresponding to the field based on the data corresponding to the field; and a second construction submodule, used to split all field values ​​using preset symbols and construct the text corresponding to the field based on the field and all the split field values.

[0017] Furthermore, the import device also includes: a first judgment module, used to determine whether the data volume of the text exceeds a preset data volume before creating a data job for each text; a first splitting module, used to split the text into multiple sub-texts if the data volume of the text exceeds the preset data volume, and each sub-text is represented as a new text; and a first creation module, used to create a data job for each new text and each text that does not exceed the preset data volume.

[0018] Furthermore, the import device also includes: a first traversal module, used to traverse the fields corresponding to each text and determine the field length of the currently traversed field before using distributed nodes to asynchronously execute data operations and import the text into a preset data table in a preset database; and a first creation module, used to create a data table associated with the field in the preset database based on the field and the field length.

[0019] Furthermore, the execution unit includes: a first query module, used to query a matching preset data table from a preset database based on the fields corresponding to the data job; and a first import module, used to import the text of the data job instruction into the preset data table using distributed nodes.

[0020] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any of the above-described methods for importing data files.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described data file import methods.

[0022] In this invention, the application file of the target application is obtained, the application file is processed to obtain multiple texts, a data job is created for each text, and for each text, the data job is executed asynchronously by distributed nodes to import the text into a preset data table in a preset database, thereby solving the technical problem of slow data file import speed in related technologies.

[0023] This invention employs intelligent analysis of application files of the target application. By scanning and identifying data files, it achieves rapid parsing and processing of various data types, thereby realizing the technical effect of efficiently importing file data into the database. This solves the technical problems of low data import efficiency and complex technical adaptation caused by the diversity of data formats and the large amount of data in related technologies. Specifically, the application files generated by the target application during its operation can be obtained first, and then these files can be processed to convert data in different fields into multiple corresponding texts. A data job is created for each text. For small data volumes, data is directly imported. For large data volumes, distributed nodes are used to asynchronously execute data jobs, importing multiple text data into a preset database in parallel. This effectively improves the processing speed of large-scale data and the efficiency of database import, reduces the workload of developers, and enhances the flexibility and responsiveness of data management. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for importing data files is shown.

[0026] Figure 2 This is a flowchart of a data file import method according to Embodiment 1 of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional data file import device according to an embodiment of the present invention;

[0028] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that all related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary confidentiality measures have been taken. These measures do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or institutions. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or institution through the interface. After receiving consent from the aforementioned user or institution, the relevant information is obtained. If the user chooses to refuse, the process enters the expert decision-making process. Data collection follows the principle of minimum necessity, processing only fields directly related to the target application; financial data is encrypted and stored, and access requires multiple authorizations.

[0032] This invention proposes an intelligent method for importing file data into a database. This method can intelligently import file data into a database and can perform database entry for data files of different types and with different data fields. This achieves rapid import of database data, reduces the development time and effort of developers, and improves their efficiency.

[0033] The present invention will now be described in detail with reference to various embodiments.

[0034] Example 1

[0035] According to an embodiment of this application, an embodiment of a data file import method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for importing data files is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera, wherein the network interface can be connected to wired and / or wireless networks. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data file import method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data file import method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0040] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0041] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for importing data files is shown. Figure 2 This is a flowchart of a data file import method according to Embodiment 1 of the present invention, as follows: Figure 2 As shown, the method includes the following steps:

[0042] Step S201: Obtain the application file of the target application, wherein the application file records the data generated during the operation of the target application.

[0043] In this embodiment of the invention, by interacting with the target application, various data files generated by the application during its operation can be captured and exported. Here, "application files" can be log files, tables, CSV (Comma-Separated Values) files, or other structured or unstructured data storage formats. The data capture process ensures coverage of key data points during application runtime, such as transaction records, user behavior data, and system logs.

[0044] Here, the target application refers to any financial application that generates data and needs to import that data into a database. Application files are files generated by the target application and containing its runtime data; these may be stored on a local hard drive, network shared space, or cloud server.

[0045] Step S202: Process the application file to obtain multiple text files, where each text file corresponds to a field.

[0046] In this embodiment of the invention, data is extracted using appropriate technologies based on the data type of the application file. For non-editable files, such as PDF (Portable Document Format) or image formats, Optical Character Recognition (OCR) technology is used for scanning and conversion, transforming data in specific fields into plain text format. For editable files, such as tables or CSV files, programming libraries and APIs (Application Programming Interfaces) are used to directly read and separate field data, forming independent text files. These text files are categorized by field for easier subsequent operations.

[0047] Here, "text" refers to the independent text files formed by breaking down the data in the application file. Each file focuses on a single data field, making it easy to manage. A field is a unit in the data file used to describe a specific aspect of information, such as a user identifier, transaction amount, or date.

[0048] Step S203: Create a data job for each text.

[0049] In this embodiment of the invention, a data job is generated for each text document. This job contains all the information required for importing the data, such as the data format, target database location, and preset data table structure. The creation process of the data job also includes preliminary verification and cleaning of the text data to ensure the accuracy and consistency of the data when imported into the database.

[0050] Here, a data job is a program or script task that includes import logic and parameters, and is used to guide how to transform and store text data into a database.

[0051] Step S204: For each text, a data job is executed asynchronously using distributed nodes to import the text into a preset data table in a preset database. The preset data table is a data table built based on the fields corresponding to the text.

[0052] In this embodiment of the invention, a distributed architecture is used to allocate data jobs to multiple computing nodes in the network. Each node executes the data job independently and in parallel, importing data into the corresponding data table of a preset database. This process can utilize multithreading or message queues to achieve asynchronous processing, ensuring efficient completion of data import even under unstable network conditions or limited computing resources.

[0053] Here, distributed nodes refer to multiple independent computing units in the system architecture used to handle data import tasks. Asynchronous execution allows data jobs to run independently without blocking other tasks, improving overall system performance. Preset databases and tables refer to databases and tables whose structure and storage strategies have been set before data import. They can be automatically constructed based on the acquired field information to adapt to the format of the data to be imported.

[0054] In summary, by employing intelligent analysis of the target application's files and scanning and identifying data files, this approach achieves rapid parsing and processing of various data types. This enables efficient import of file data into the database, resolving the technical challenges of low import efficiency and complex technical adaptation caused by diverse data formats and massive data volumes. Specifically, the approach first acquires the application files generated during the target application's runtime, then processes these files, converting data from different fields into multiple corresponding text files. A data job is created for each text file. For smaller datasets, data is directly imported. For large datasets, distributed nodes asynchronously execute data jobs, importing multiple text files in parallel into a pre-defined database. This effectively improves the processing speed and database import efficiency of large-scale data, reduces the workload of developers, and enhances the flexibility and responsiveness of data management.

[0055] To improve the accuracy of processing application files, in the data file import method provided in Embodiment 1 of this application, the file type of the application file is determined, wherein the file type includes: image file type and text file type; when the file type of the application file is an image file type, the application file is scanned to obtain scan data, and the scan data is processed to obtain multiple texts; when the file type of the application file is a text file type, the application file is processed to obtain multiple texts.

[0056] In this embodiment of the invention, the file type of the application file can be determined first. This identification process is achieved by analyzing the file extension or its metadata. Here, "file type" refers to the format of the application file, which mainly includes: image file types (such as non-editable files like PDF) and text file types (such as editable files like CSV and TXT).

[0057] When the application file is an image file, Optical Character Recognition (OCR) technology can be used to scan the file and convert the text information in the image into editable text. After scanning, the resulting scanned data is raw text containing the text content of the application file, but the data in the text is still messy and requires further processing. Then, the scanned data is processed using natural language processing techniques or regular expressions to identify and segment the data corresponding to different fields. This processing decomposes the scanned data into multiple texts, each focusing on a single data field, such as date, amount, or user identifier. This fine-grained file classification simplifies subsequent data import operations and ensures the independence and integrity of each field's data.

[0058] When the application file is a text file, the file content is read directly. Using programming libraries and APIs, the file structure and field information can be accurately identified. The entire application file is then processed to separate data associated with different fields, generating multiple text files. This allows for direct location and extraction of the required data, avoiding the complex image analysis and conversion required in image file processing, and improving data processing efficiency.

[0059] In this embodiment, the system can intelligently identify and adapt different types of input files, whether image or text files, effectively converting them into multiple text files suitable for database import. This process not only reduces pre-processing time but also improves import accuracy because the data is precisely categorized and cleaned, reducing the complexity of subsequent data validation and processing. Simultaneously, it enhances system flexibility, eliminating the need for developers to write specific import scripts for each file format, saving development resources. Whether for small-scale data imports or large-scale data migrations, it provides stable and efficient data processing services, offering robust support for rapid application deployment and data management.

[0060] To improve the accuracy of processing scanned data, the data file import method provided in Embodiment 1 of this application extracts multiple fields from the scanned data and obtains the data corresponding to each field from the scanned data; based on the data corresponding to the fields, determines all field values ​​corresponding to the fields; and based on the fields and all field values ​​corresponding to the fields, constructs the text corresponding to the fields.

[0061] In this embodiment of the invention, scanned data refers to raw text data obtained after performing OCR processing on an application file of image file type. This data contains a large amount of information but has not yet been structured, and therefore may appear in an unorganized form. To make the scanned data useful, it must be further analyzed to extract multiple fields and obtain the specific data corresponding to each field. For example, key information units in the data, such as dates, numbers, and names, can be identified and classified using NLP (Natural Language Processing) technology or pattern recognition algorithms.

[0062] After extracting and retrieving the field data, all possible field values ​​for each field can be determined. This can be achieved by collecting every instance of a field. Then, based on the determined fields and all their values, a series of texts corresponding to each field is constructed. These texts are carefully designed to conform to database import standards; for example, specific delimiters and formats are used to organize the field values ​​so that the database program can correctly identify and store them. The process of constructing the text also includes checking the maximum length of the field values ​​to accommodate the database field width limitations.

[0063] For example, once the scanned data is complete, natural language processing techniques are used, such as keyword detection, regular expression matching, or machine learning models, to identify potential fields, such as identifying words like "date," "amount," or similar markers as clues. Once a field is identified, all information instances related to that field are extracted from the scanned data, i.e., the data corresponding to each field is obtained. This process may involve searching, filtering, and sorting text fragments to ensure that the dataset corresponding to each field is complete. Then, the dataset is further analyzed to determine all field values ​​corresponding to each field, including the data distribution, format, and potential anomalies. For example, if a field is a date, the consistency of the date format needs to be verified. Based on this information, the text corresponding to each field is constructed, ensuring that each text contains data for only a single field while meeting the size and format requirements of the database fields. This construction work may involve data transformation, standardization, and population to fit the database structure.

[0064] In this embodiment, the chaotic state of scanned data can be effectively transformed into a well-ordered system, achieving efficient conversion from image files to structured database data. This not only improves the automation of data import and reduces the need for manual intervention, but also ensures the accuracy and consistency of the data, laying a solid foundation for subsequent data analysis and applications.

[0065] To improve the accuracy of processing text-based application files, the data file import method provided in Embodiment 1 of this application extracts text data from the application file and identifies the text data to obtain multiple fields; obtains the data corresponding to each field from the text data, and determines all field values ​​corresponding to the field based on the data corresponding to the field; uses a preset symbol to split all field values, and constructs the text corresponding to the field based on the field and all the split field values.

[0066] In this embodiment of the invention, for an editable file (such as a document), a corresponding tool is used to generate a text for all the data corresponding to each field. This text separates all the values ​​in order using special symbols and records the field length of the longest value among all the values ​​corresponding to each field.

[0067] Specifically, the text data of editable application files can be extracted first. Natural Language Processing (NLP) techniques and machine learning models are then used to recognize the extracted text data, automatically identifying and separating meaningful information blocks from the text, such as customer ID, transaction date, and amount. These information blocks are the fields. After identifying the fields, specific data values ​​directly associated with each field can be extracted from the text data. For example, if "date" is identified as a field, all possible text fragments representing dates can be filtered out. These data values ​​are further analyzed to determine all possible values ​​covered under each field. Furthermore, to facilitate data import and processing, pre-defined special symbols (such as commas or vertical bars) are used to separate all field values ​​for each field. This segmentation method provides clear data boundary markers, ensuring that data is accurately identified and placed in the correct field positions when imported into the database. Then, based on the fields and their segmented field values, a series of text files are constructed. Each text file contains all data values ​​for only one field, arranged according to the preset delimiters. Such text files fully conform to the database import format and can be directly used for data import operations.

[0068] In this embodiment, editable unstructured application file data is converted into a structured, database-importable text format. The entire process is highly automated, reducing manual intervention and saving time and labor costs. Intelligent recognition and data extraction ensure data accuracy and integrity, avoiding common errors in data processing. Preset symbols are used to separate field values, ensuring a clear data structure that is easy for the database management system to understand and store. The final text is directly applicable to the database import process, improving the efficiency and convenience of data import.

[0069] In order to accurately process large amounts of text, in the data file import method provided in Embodiment 1 of this application, before creating a data job for each text, it is determined whether the data volume of the text exceeds a preset data volume; if the data volume of the text exceeds the preset data volume, the text is split into multiple sub-texts, and each sub-text is represented as a new text; a data job is created for each new text and each text that does not exceed the preset data volume.

[0070] In this embodiment of the invention, when processing large amounts of data, a preliminary check is performed on the text to be imported into the database to determine whether it exceeds a preset data volume (i.e., file size or number of records). This preset data volume is a threshold set based on factors such as the performance, storage capacity, and concurrent processing capabilities of the target database, which optimizes data import efficiency and avoids excessive resource consumption by a single import operation. If the text data volume exceeds the preset data volume, a splitting operation can be performed to divide the original text into multiple sub-texts. This splitting process typically considers data consistency, integrity, and ease of subsequent processing. For example, it ensures that each sub-text file contains all necessary fields and that the data is divided logically or chronologically to maintain data continuity. Each sub-text is treated as a new text after splitting. This facilitates subsequent independent processing, as each new text file is managed as an independent data import task, thereby achieving parallelization and efficiency in data import. Whether it is a new text after splitting or a text whose original data volume does not exceed the preset data volume, each part will be created as an independent data job. Each data job contains complete import instructions, target database information, and the text data to be imported. By creating these data jobs, the import process for each file can be effectively managed and scheduled, ensuring smooth operation even when processing large amounts of data and avoiding the risk of database system overload.

[0071] In this embodiment, efficient and orderly import is achieved when processing large amounts of text. This not only improves the speed of data import but also reduces the pressure on the database system, effectively avoiding performance bottlenecks that may be encountered when importing large datasets. Furthermore, by splitting the data into multiple sub-texts and creating corresponding data jobs, the flexibility and manageability of the data import process are enhanced, allowing it to better adapt to different database configurations and network environments, thus achieving optimal data import results in various scenarios.

[0072] In order to accurately construct the data table, in the data file import method provided in Embodiment 1 of this application, before the data job is executed asynchronously by distributed nodes and the text is imported into the preset data table in the preset database, the fields corresponding to each text are traversed and the field length of the currently traversed field is determined. The field length is the length of the longest field value among all the field values ​​corresponding to the field. Based on the field and the field length, a data table associated with the field is established in the preset database.

[0073] In this embodiment of the invention, the fields corresponding to each text can be traversed. During the traversal, the field length of each field can be calculated. The field length refers to the length of the longest value among all field values ​​under that field. This ensures that the database fields can accommodate all possible data values, avoiding data truncation or storage failures caused by improper field length settings. After obtaining the length information of all fields, a data table is constructed in a preset database based on this information. The structure of each table is closely related to the fields in the text, ensuring that each field can find a corresponding storage location in the database table, and that the width of that location is sufficient to accommodate the longest field value under that field.

[0074] In this embodiment, the system can automatically adapt to text with different formats and data volumes, intelligently constructing the database table structure. By traversing fields and accurately calculating field lengths, it ensures that the constructed data table perfectly matches the data in the text, thereby improving the accuracy and efficiency of data import. Simultaneously, it enhances the system's flexibility, enabling rapid adjustments to the database table structure to meet data storage needs even in the face of frequent changes in data format and field types.

[0075] To improve the accuracy of asynchronous data job execution, in the data file import method provided in Embodiment 1 of this application, a matching preset data table is queried from a preset database based on the fields corresponding to the data job; and distributed nodes are used to import the text of the data job instruction into the preset data table.

[0076] In this embodiment of the invention, before data import, a matching data table can be automatically searched in a preset database using data job instructions. Here, a data job refers to a detailed set of instructions for each text import operation, including field information, data file path, target database information, etc. By comparing the fields mentioned in the data job with the fields of existing data tables in the preset database, the correct data table is determined, thus ensuring that the data can be correctly imported to the intended location. After the data table is determined, distributed node technology is used to process the data import task in parallel. Distributed node technology allows the execution of data jobs to be distributed across multiple servers or computing nodes, rather than concentrated on a single node. This technology fully utilizes the processing power of multiple servers, significantly reducing the total data import time, especially for large-volume data import tasks. The data job is decomposed into multiple subtasks, each subtask processing a portion of the text or a subset of the text, and then these data are imported in parallel into the corresponding data tables in the preset database.

[0077] This embodiment effectively combines the refined management of data jobs with the high-performance computing capabilities of distributed nodes, achieving efficient and secure import of large-scale datasets. By automatically matching data jobs with database tables, the pre-import preparation is simplified, reducing the possibility of human error. Furthermore, the application of distributed node technology overcomes the performance bottleneck of traditional single-machine data import, enabling the completion of large-scale data import operations in a very short time, thus improving work efficiency.

[0078] The data file import method provided in this application embodiment employs intelligent analysis of the application files of the target application. By scanning and identifying data files, it achieves rapid parsing and processing of various data types, thereby realizing the technical effect of efficiently importing file data into the database. This solves the technical problems of low data import efficiency and complex technical adaptation caused by the diversity of data formats and the large amount of data in related technologies. Specifically, the application files generated by the target application during its operation can be obtained first, and then these files can be processed to convert data in different fields into multiple corresponding texts. A data job is created for each text. For small data volumes, data is directly imported. For large data volumes, distributed nodes are used to asynchronously execute data jobs, importing multiple text data into a preset database in parallel. This effectively improves the processing speed of large-scale data and the efficiency of database import, reduces the workload of developers, and enhances the flexibility and responsiveness of data management.

[0079] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0080] Example 2

[0081] This application also provides a data file import device. It should be noted that the data file import device of this application can be used to execute the data file import method provided in this application. The data file import device provided in this application is described below.

[0082] According to an embodiment of this application, an apparatus for implementing the above-described data file import method is also provided. Figure 3 This is a schematic diagram of an optional data file import device according to an embodiment of the present invention, such as... Figure 3 As shown, the import device may include: an acquisition unit 30, a processing unit 31, a creation unit 32, and an execution unit 33.

[0083] The acquisition unit 30 is used to acquire the application file of the target application, wherein the application file records the data generated during the operation of the target application;

[0084] Processing unit 31 is used to process the application file to obtain multiple texts, where each text corresponds to a field;

[0085] Create cell 32 to create a data job for each text;

[0086] Execution unit 33 is used to asynchronously execute data jobs using distributed nodes for each text, importing the text into a preset data table in a preset database, wherein the preset data table is a data table constructed based on the fields corresponding to the text.

[0087] The data file import device provided in this application embodiment employs intelligent analysis of the application files of the target application. By scanning and identifying data files, it achieves rapid parsing and processing of various data types, thereby realizing the technical effect of efficiently importing file data into the database. This solves the technical problems of low data import efficiency and complex technical adaptation caused by the diversity of data formats and the large amount of data in related technologies. Specifically, it can first obtain the application files generated by the target application during its operation, and then process these files, converting data in different fields into multiple corresponding texts. A data job is created for each text. For small data volumes, data is directly imported. For large data volumes, distributed nodes are used to asynchronously execute data jobs, importing multiple text data into a preset database in parallel. This effectively improves the processing speed of large-scale data and the efficiency of database import, reduces the workload of developers, and enhances the flexibility and responsiveness of data management.

[0088] Optionally, the processing unit includes: a first determining module, used to determine the file type of the application file, wherein the file type includes: image file type and text file type; a first processing module, used to scan the application file when the file type of the application file is image file type, obtain scan data, and process the scan data to obtain multiple texts; and a second processing module, used to process the application file when the file type of the application file is text file type to obtain multiple texts.

[0089] Optionally, the first processing module includes: a first extraction submodule, used to extract multiple fields from the scanned data and obtain the data corresponding to each field from the scanned data; a first determination submodule, used to determine all field values ​​corresponding to the field based on the data corresponding to the field; and a first construction submodule, used to construct the text corresponding to the field based on the field and all field values ​​corresponding to the field.

[0090] Optionally, the second processing module includes: a second extraction submodule, used to extract text data from the application file and recognize the text data to obtain multiple fields; a second determination submodule, used to obtain the data corresponding to each field from the text data and determine all field values ​​corresponding to the field based on the data corresponding to the field; and a second construction submodule, used to split all field values ​​using a preset symbol and construct the text corresponding to the field based on the field and all the split field values.

[0091] Optionally, the import device further includes: a first judgment module, used to determine whether the data volume of the text exceeds a preset data volume before creating a data job for each text; a first splitting module, used to split the text into multiple sub-texts if the data volume of the text exceeds the preset data volume, and each sub-text is represented as a new text; and a first creation module, used to create a data job for each new text and each text that does not exceed the preset data volume.

[0092] Optionally, the import device further includes: a first traversal module, used to traverse the fields corresponding to each text and determine the field length of the currently traversed field before using distributed nodes to asynchronously execute data operations and import the text into a preset data table in a preset database; and a first creation module, used to create a data table associated with the field in the preset database based on the field and the field length.

[0093] Optionally, the execution unit includes: a first query module, used to query a matching preset data table from a preset database based on the fields corresponding to the data job; and a first import module, used to import the text of the data job instruction into the preset data table using distributed nodes.

[0094] The aforementioned import device may also include a processor and a memory. The aforementioned acquisition unit 30, processing unit 31, creation unit 32, execution unit 33, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0095] The aforementioned processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and by adjusting kernel parameters, data jobs are executed asynchronously on distributed nodes for each text file, importing the text into a preset data table in a preset database.

[0096] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0097] It should be noted that the acquisition unit 30, processing unit 31, creation unit 32, and execution unit 33 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0098] Example 3

[0099] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0100] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data file import method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned data file import method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtain the application file of the target application, wherein the application file records data generated during the operation of the target application; process the application file to obtain multiple texts, wherein each text corresponds to a field; create a data job for each text; for each text, use distributed nodes to asynchronously execute the data job to import the text into a preset data table in a preset database, wherein the preset data table is a data table built based on the field corresponding to the text.

[0102] The processor can access information and applications stored in memory via a transmission device to perform the following steps: determine the file type of the application file, where the file type includes: image file type and text file type; if the application file is an image file type, scan the application file to obtain scan data, and process the scan data to obtain multiple text files; if the application file is a text file type, process the application file to obtain multiple text files.

[0103] The processor can access information and applications stored in memory via a transmission device to perform the following steps: extract multiple fields from the scanned data and obtain the data corresponding to each field from the scanned data; determine all field values ​​corresponding to the field based on the data corresponding to the field; and construct the text corresponding to the field based on the field and all field values ​​corresponding to the field.

[0104] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: extract text data from the application file and recognize the text data to obtain multiple fields; obtain the data corresponding to each field from the text data and determine all field values ​​corresponding to the field based on the data corresponding to the field; split all field values ​​using a preset symbol and construct the text corresponding to the field based on the field and all the split field values.

[0105] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: determine whether the amount of text data exceeds a preset amount of data; if the amount of text data exceeds the preset amount of data, split the text into multiple sub-texts, and represent each sub-text as a new text; create a data job for each new text and each text that does not exceed the preset amount of data.

[0106] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: traversing the fields corresponding to each text and determining the field length of the currently traversed field, where the field length is the length of the longest field value among all field values ​​corresponding to the field; based on the field and the field length, establishing a data table associated with the field in a preset database.

[0107] The processor can access information and applications stored in memory via a transmission device to perform the following steps: query a matching preset data table from a preset database based on the fields corresponding to the data job; and import the text of the data job instruction into the preset data table using distributed nodes.

[0108] This application provides a data file import scheme. By employing intelligent analysis of the target application's files, and through scanning and identifying data files, it achieves rapid parsing and processing of various data types. This enables efficient import of file data into a database, thus solving the technical problems of low data import efficiency and complex technical adaptation caused by the diversity of data formats and the large volume of data in related technologies. Specifically, the application files generated during the target application's operation are first obtained, and then these files are processed. Data from different fields is converted into multiple corresponding text files, and a data job is created for each text file. For small data volumes, data is directly imported. For large data volumes, distributed nodes asynchronously execute data jobs, importing multiple text data files in parallel into a preset database. This effectively improves the processing speed of large-scale data and the efficiency of database import, reduces the workload of developers, and enhances the flexibility and responsiveness of data management.

[0109] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be terminal devices such as smartphones, tablets, PDAs, and mobile internet devices (MIDs). Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0110] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0111] Example 4

[0112] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data file import method provided in Embodiment 1.

[0113] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0114] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of a data file import method.

[0115] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0116] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0121] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for importing data files, characterized in that, include: Obtain the application file of the target application, wherein the application file records the data generated during the operation of the target application; The application file is processed to obtain multiple text files, each of which corresponds to a field; Create a data job for each of the texts; For each text, the data job is executed asynchronously using distributed nodes to import the text into a preset data table in a preset database, wherein the preset data table is a data table constructed based on the fields corresponding to the text.

2. The import method according to claim 1, characterized in that, The steps of processing the application file to obtain multiple text files include: The file type of the application file is determined, wherein the file type includes: image file type and text file type; If the file type of the application file is the image file type, the application file is scanned to obtain scan data, and the scan data is processed to obtain multiple text files; If the file type of the application file is a text file type, the application file is processed to obtain multiple text files.

3. The import method according to claim 2, characterized in that, The step of processing the scanned data to obtain multiple texts includes: Extract multiple fields from the scan data, and obtain the data corresponding to each field from the scan data; Based on the data corresponding to the field, determine the values ​​of all fields corresponding to the field; Based on the field and all the field values ​​corresponding to the field, construct the text corresponding to the field.

4. The import method according to claim 2, characterized in that, The steps of processing the application file to obtain multiple texts include: Extract the text data from the application file and recognize the text data to obtain multiple fields; Obtain the data corresponding to each field from the text data, and determine the values ​​of all fields corresponding to the field based on the data corresponding to the field; All field values ​​are separated using a preset symbol, and the text corresponding to the field is constructed based on the field and all the separated field values.

5. The import method according to claim 1, characterized in that, Before creating a data job for each of the texts, the following is also included: Determine whether the amount of data in the text exceeds a preset data amount; If the amount of data in the text exceeds the preset amount of data, the text is split into multiple sub-texts, and each sub-text is represented as a new text. Create a data job for each new text and for each text that does not exceed the preset data amount.

6. The import method according to claim 1, characterized in that, Before the data job is executed asynchronously using distributed nodes and the text is imported into a preset data table in a preset database, the process further includes: Iterate through the fields corresponding to each text and determine the field length of the currently iterated field, wherein the field length is the length of the longest field value among all field values ​​corresponding to the field; Based on the field and the field length, a data table associated with the field is created in the preset database.

7. The import method according to claim 1, characterized in that, The step of asynchronously executing the data job using distributed nodes to import the text into a preset data table in a preset database includes: Based on the fields corresponding to the data job, query the matching preset data table from the preset database; The distributed nodes are used to import the text of the data job instruction into the preset data table.

8. A data file import device, characterized in that, include: The acquisition unit is used to acquire the application file of the target application, wherein the application file records the data generated during the operation of the target application; The processing unit is used to process the application file to obtain multiple texts, wherein each text corresponds to a field; Create a unit to create a data job for each of the texts; An execution unit is configured to asynchronously execute the data job for each text using distributed nodes, importing the text into a preset data table in a preset database, wherein the preset data table is a data table constructed based on the fields corresponding to the text.

9. A computer program product, characterized in that, The method includes a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the data file import method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data file import method according to any one of claims 1 to 7.

Images