Data processing method, device, processor and electronic device

Abstract

The present application discloses a data processing method, device, processor and electronic device. It relates to the field of cloud computing, and the method includes: obtaining a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and multiple table fields in the database table respectively correspond to the file positions of the file to be loaded; cutting the file to be loaded according to the configuration file, and determining multiple file fields corresponding to multiple table fields; processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; and storing the processed file fields in the corresponding database. Through the present application, the problem of poor stability and low efficiency of the distributed system in the related art when loading large quantities of data is solved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing, and more specifically, to a data processing method, device, processor, and electronic device. Background Art

[0002] The current Internet industry is developing rapidly, and the demand for data sharing and transmission between enterprises, products, and applications is also growing. In an era of skyrocketing information volume, when loading large quantities of data files, it is also necessary to ensure the stability of the database and ensure that the performance of external online services is not affected by data loading, resulting in a decline in external service experience. At the same time, on the open platform system, there are hundreds of database systems, and there is a certain degree of heterogeneity between databases, resulting in data file loading solutions that are incompatible with various database systems, and the replacement or docking of database systems will bring too much system migration workload.

[0003] Distributed systems in related technologies have problems of poor stability and low efficiency when loading large quantities of data, and no effective solution has been proposed yet. Summary of the invention

[0004] The main purpose of the present application is to provide a data processing method, device, processor and electronic device to solve the problems of poor stability and low efficiency in the distributed system in the related art when loading large amounts of data.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided, comprising: obtaining a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and multiple table fields in the database table respectively correspond to the file positions of the file to be loaded; according to the configuration file, the file to be loaded is trimmed to determine multiple file fields corresponding to the multiple table fields; according to the processing function corresponding to the table field, the file field corresponding to the table field is processed; and the processed file field is stored in the corresponding database.

[0006] Optionally, before the files to be loaded are trimmed according to the configuration file and the multiple file fields corresponding to the multiple table fields are determined, the method further includes: obtaining a file list of the files to be loaded, wherein the file list includes the file names of the multiple files to be loaded; parsing the configuration file to determine the file names of the files to be loaded and the database table name, wherein the multiple table fields in the database table respectively correspond to the file locations of the files to be loaded; and traversing the file list to check the multiple files to be loaded.

[0007] Optionally, traversing the file list and checking the multiple files to be loaded include: traversing the file list and reading the length of the file line of the file to be loaded; determining the length of the file line corresponding to the table field of the database table according to the configuration file; when the length of the read file line is consistent with the length of the file line corresponding to the database table, executing the step of trimming the file to be loaded according to the configuration file to determine the multiple file fields corresponding to the multiple table fields; when the length of the read file line is inconsistent with the length of the file line corresponding to the database table, skipping the file line of the file to be loaded, checking subsequent file lines, and logging the skipped file line.

[0008] Optionally, the file name of the file to be loaded, and the database table name, and the multiple table fields in the database table respectively correspond to the file position of the file to be loaded, including: according to the file position corresponding to the table field, cropping the content data in the file to be loaded at the file position; when the cropped content data is not empty, using the cropped content data as the file field, establishing a corresponding relationship with the table field, and storing it; when the cropped content data is empty, using the preset default content data as the file field, establishing a corresponding relationship with the table field, and storing it.

[0009] Optionally, processing the file field corresponding to the table field according to the processing function corresponding to the table field includes: determining the processing function corresponding to the table field in the corresponding relationship according to the corresponding relationship, wherein the processing function corresponding to the table field is pre-set in the configuration file; and processing the file field corresponding to the corresponding relationship through the processing function.

[0010] Optionally, storing the processed file field in the corresponding database includes: when the processing function is a built-in function, processing the file field locally through the built-in function; when the processing function is a non-built-in function, sending the file field to the database, wherein the database processes the file field through the non-built-in function before loading the file field.

[0011] Optionally, storing the processed file fields in a corresponding database includes: determining a loading script corresponding to the database according to the processed file fields; and storing the file fields in the database by running the loading script.

[0012] Optionally, there are multiple databases, and before processing the file field corresponding to the table field according to the processing function corresponding to the table field, the method also includes: according to the identification field in the file to be loaded, wherein the identification field is used to identify the database corresponding to the file field in the file to be loaded; according to the identification field and the file field identified by the identification field, determining the database identifier stored in the file field.

[0013] Optionally, before storing the processed file fields in the corresponding database, the method further includes: recording the number of file lines in which processing has been completed in the file to be loaded; and when the number of file lines reaches a preset number, storing the processed file fields in the file to be loaded in the corresponding database.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data processing device is provided, including: an acquisition module, used to acquire a file to be loaded, and a configuration file, wherein the configuration file includes the file name of the file to be loaded, and the table name of a database table, and multiple table fields in the database table respectively correspond to the file positions of the file to be loaded; a trimming module, used to trim the file to be loaded according to the configuration file, and determine multiple file fields corresponding to the multiple table fields; a processing module, used to process the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; and a storage module, used to store the processed file fields in the corresponding database.

[0015] In order to achieve the above objective, according to another aspect of the present application, a processor is provided, wherein the processor is used to run a program, wherein the program executes any one of the data processing methods described above when running.

[0016] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods described above.

[0017] Through this application, the following steps are adopted: obtaining the file to be loaded and the configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of the database table, and the multiple table fields in the database table correspond to the file position of the file to be loaded; cutting the file to be loaded according to the configuration file, and determining the multiple file fields corresponding to the multiple table fields; processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; and storing the processed file fields in the corresponding database. The relationship between the file to be loaded and the table fields of the database table is configured through the configuration file, and the file to be loaded is loaded into the database after processing according to the processing functions corresponding to the table fields, which solves the problem of poor stability and low efficiency of the distributed system in the related art when loading large batches of data. Thus, the large batches of data files can be loaded smoothly and quickly, and the impact on the database performance is small, which improves the stability and loading efficiency of the distributed system in large batches of data loading. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a batch file loading system provided according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of batch file loading provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a data processing device provided according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0028] Step S101, obtaining a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and a plurality of table fields in the database table respectively correspond to the file locations of the file to be loaded;

[0029] Step S102, cutting the file to be loaded according to the configuration file, and determining a plurality of file fields corresponding to a plurality of table fields;

[0030] Step S103, processing the file field corresponding to the table field according to the processing function corresponding to the table field;

[0031] Step S104, storing the processed file fields in the corresponding database.

[0032] The execution subject of the above steps can be a processor or a controller. Through the above steps, the file to be loaded and the configuration file are obtained, wherein the configuration file includes the file name of the file to be loaded and the table name of the database table, and the multiple table fields in the database table correspond to the file position of the file to be loaded respectively; the file to be loaded is cut according to the configuration file, and the multiple file fields corresponding to the multiple table fields are determined; the file fields corresponding to the table fields are processed according to the processing functions corresponding to the table fields; and the processed file fields are stored in the corresponding database. The relationship between the file to be loaded and the table fields of the database table is configured through the configuration file, and the file to be loaded is loaded into the database after being processed according to the processing functions corresponding to the table fields, which solves the problem of poor stability and low efficiency of the distributed system in the related technology when loading large batches of data. Then, the large batch of data files are loaded smoothly and quickly, and the impact on the database performance is small, which improves the stability and loading efficiency of the distributed system in large batches of data loading.

[0033] The file to be loaded can be a file for data migration between different database systems in a distributed system. In the related art, there are not only differences in versions of database systems, but also different structures of database systems for different developers. When migrating data or replacing systems between different database systems, there is a large amount of file migration. During loading, due to the large amount of data, it often causes database instability and affects the performance of the database. This embodiment provides a loading method, which loads the loading file through a configuration file and a database table. The processing of the file to be loaded can be matched directly through the database table, and then loaded. This avoids the problem in the prior art that the file to be loaded is loaded into the database and then processed, which will occupy the resources of the database and affect the performance of the database, and improves the stability and loading efficiency of the file to be loaded to a certain extent.

[0034] The above configuration file includes the file name of the file to be loaded and the table name of the database table. The multiple table fields in the database table correspond to the file positions of the files to be loaded respectively. It can also include the default value of each table field, that is, the table field corresponds to the file position of the file to be loaded. When determining the corresponding file content, if the corresponding file content is empty, it is replaced with the default value to avoid the table field having no corresponding content, which easily causes interference to other corresponding relationships. The above configuration file can also include the processing method corresponding to each table field, so that the file content corresponding to the table field can be processed according to the preset processing method in the future.

[0035] The above database table includes multiple table fields, each of which corresponds to the file position of the file to be loaded. It should be noted that the above file to be loaded includes multiple rows, each of which corresponds to the position of a row. Since the file to be loaded stores data in rows, a table field only corresponds to the file content from the first position to the second position in a row, and there is no file content where a table field corresponds to multiple rows.

[0036] The above method cuts the file to be loaded according to the configuration file to determine multiple file fields corresponding to multiple table fields, and then stores the file fields corresponding to the table fields in the location of the database table, or stores the storage location of the file fields through the table fields, so that the file fields can be processed and loaded later.

[0037] According to the processing function corresponding to the table field, the file field corresponding to the table field is processed. The processing function can be a sum function, a rounding function, and other functions for processing fields. The processing function can be divided into built-in functions of the database table and non-built-in functions. Non-built-in functions are processing functions provided by services other than the database table. The processing functions corresponding to the file fields corresponding to multiple table fields can be the same.

[0038] After being processed by the processing function, the processed file fields are stored in the corresponding database. The above file fields include a field for identifying the database, and the database corresponding to the file field can be determined through this field. Thus, the relationship between the file to be loaded and the table field of the database table is configured through the configuration file, and the file to be loaded is loaded into the database after being processed according to the processing function corresponding to the table field, which solves the problem of poor stability and low efficiency of the distributed system in the related technology when loading large quantities of data. Thus, the large batch of data files can be loaded smoothly and quickly, and the impact on the database performance is small, which improves the stability and loading efficiency of the distributed system in large batches of data.

[0039] Optionally, before the file to be loaded is trimmed according to the configuration file and multiple file fields corresponding to multiple table fields are determined, the method also includes: obtaining a file list of the files to be loaded, wherein the file list includes the file names of the multiple files to be loaded; parsing the configuration file to determine the file name of the file to be loaded and the database table name, wherein the multiple table fields in the database table respectively correspond to the file locations of the files to be loaded; traversing the file list and checking the multiple files to be loaded.

[0040] The above-mentioned file list can be a file list determined by the loading requirements, including the file names of multiple files to be loaded. The configuration file is parsed to obtain the file name of the file to be loaded corresponding to the configuration file, that is, the file to be loaded corresponding to the configuration file adopts the configuration file. And the database table name, that is, the database table used when the file to be loaded corresponding to the configuration file is loaded, and multiple table fields in the database table are set to correspond to the file positions of the files to be loaded.

[0041] Optionally, traversing the file list and checking multiple files to be loaded include: traversing the file list and reading the length of the file line of the file to be loaded; determining the length of the file line corresponding to the table field of the database table according to the configuration file; when the length of the read file line is consistent with the length of the file line corresponding to the database table, executing the step of trimming the file to be loaded according to the configuration file and determining multiple file fields corresponding to the multiple table fields; when the length of the read file line is inconsistent with the length of the file line corresponding to the database table, skipping the file line of the file to be loaded, checking subsequent file lines, and logging the skipped file lines.

[0042] Since the table field of the above database table corresponds to the file name of a row of the file to be loaded, before obtaining the file field of the file to be loaded corresponding to the table field, the file line length set in the database table for the row of the file can be determined by corresponding to multiple table fields of the same row, and compared with the length of the file line of the file to be loaded. If the lengths are consistent, it means that there is no problem with the file to be loaded and it can be trimmed. If the lengths are inconsistent, it means that there may be an error in the file field of the file to be loaded, and a log can be registered or an error can be reported. This ensures the accuracy of reading the file to be loaded, and then improves the loading accuracy of the file to be loaded.

[0043] Optionally, the file name of the file to be loaded, and the database table name, and multiple table fields in the database table respectively correspond to the file position of the file to be loaded, including: according to the file position corresponding to the table field, the content data at the file position in the file to be loaded is cropped; when the cropped content data is not empty, the cropped content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed; when the cropped content data is empty, the preset default content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed.

[0044] The file position of the file to be loaded corresponding to the table field is empty, indicating that the file to be loaded has no content at this position. In order to prevent the stability and accuracy of the correspondence between the table field and the file field of the file to be loaded, the file position without the file field can be filled with the default value to establish a correspondence with the table field. When loading the processed file field into the database, the default field can be deleted to accurately load the file to be loaded and ensure the consistency of the file to be loaded before and after loading. This further improves the loading accuracy and stability of the file to be loaded.

[0045] Optionally, processing the file field corresponding to the table field according to the processing function corresponding to the table field includes: determining the processing function corresponding to the table field in the corresponding relationship according to the corresponding relationship, wherein the processing function corresponding to the table field is pre-set in the configuration file; and processing the file field corresponding to the corresponding relationship through the processing function.

[0046] The processing function corresponding to the table field can be included in the configuration file. By parsing the configuration file, the processing function corresponding to the table field of the database table can be determined. The processing function corresponding to the table field can also be pre-set and can be modified according to needs during use.

[0047] Optionally, storing the processed file field in the corresponding database includes: when the processing function is a built-in function, processing the file field locally through the built-in function; when the processing function is a non-built-in function, sending the file field to the database, wherein the database processes the file field through the non-built-in function before loading the file field.

[0048] When the processing function is a built-in function, that is, the processing function that the database table has, the file field corresponding to the table field can be processed in the database table to avoid processing the file field during the file loading process, which will occupy more system resources. For non-built-in functions, the processing function that the database table does not have, the file field needs to be processed during the process of loading the file field into the database, and then the processed file field is loaded into the corresponding database.

[0049] Optionally, storing the processed file fields in a corresponding database includes: determining a loading script corresponding to the database according to the processed file fields; and storing the file fields in the database by running the loading script.

[0050] The above-mentioned database can be a distributed database, for example, a MySQL database. The above-mentioned loading script can be a SQL script required for loading the distributed database. In one embodiment, the processed file fields are assembled into different batch SQL scripts according to different loading databases. For example, for a MYSQL database, the following script format can be used. When the database record does not exist, it is inserted; if it exists, the original record is updated: INSERT INTO TABLENAME(A,B)VALUES('VALUEA','VALUEB'),('VALUEC','VALUED')ON DUPLICATE KEY UPDATE A=VALUES(A),B=VALUES(B).

[0051] Optionally, there are multiple databases, and before processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields, the method also includes: according to the identification field in the file to be loaded, wherein the identification field is used to identify the database corresponding to the file field in the file to be loaded; according to the identification field and the file field identified by the identification field, determining the database identifier stored in the file field.

[0052] The above identification field can determine the database corresponding to the file field in the file to be loaded, thereby generating a database identifier to identify the file field, so as to determine the database corresponding to the file field, which is convenient for subsequent loading.

[0053] Optionally, before storing the processed file fields in the corresponding database, the method also includes: recording the number of file lines that have been processed in the file to be loaded; when the number of file lines reaches a preset number, storing the processed file fields in the file to be loaded in the corresponding database.

[0054] In order to improve the loading efficiency of batch files, when matching the files to be loaded with the database tables, the file rows that have already corresponded to the database tables are counted. When the number reaches the preset number, it means that the file volume of the file fields to be loaded has reached a certain level. If the number is larger, the loading may be stuck or fail due to the upper limit of the system loading performance. Therefore, when the number of file rows reaches the preset number, the file fields that have already corresponded to the table fields of the database table are processed and loaded to ensure that the loading can be carried out efficiently, thereby improving the accuracy and stability of loading the files to be loaded.

[0055] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] This embodiment also provides an optional implementation mode, which is described in detail below.

[0057] This embodiment aims at the defects and deficiencies in the prior art and provides a large batch data file loading device that is stable, has little impact on database performance fluctuations, does not require additional installation of database clients, and is compatible with heterogeneous databases.

[0058] Figure 2 is a schematic diagram of a batch file loading system provided according to an embodiment of the present application, such as Figure 2 As shown, the main idea of ​​this device is to parse and split large file records through the correspondence between the configuration file and the table structure, reorganize them into small batches of loading scripts, submit them to the database for loading, and reduce the database transactions for loading.

[0059] This device is developed using Java and is ultimately packaged as a Jar package for use by users. Users can import the Jar package directly or through Maven dependencies without having to install an additional database client.

[0060] After the device is introduced, the corresponding relationship and conversion relationship between the configuration file and the data table fields are as follows:

[0061] 1. The file name and database table name to be loaded.

[0062] 2. For each table field, the starting position in the file record line.

[0063] 3. The default value of each table field. If the file value content cannot be obtained, the default value is used to load the database.

[0064] 4. The data processing method for each table field. If necessary, you can configure the data processing method for each table field, such as RTRIM to remove spaces on the right or assemble it from several fields in the file.

[0065] Figure 3 is a flowchart of batch file loading provided according to an embodiment of the present application, such as Figure 3 As shown, after the device is started, it will press Figure 3 The process shown is to load batch files.

[0066] 1. Parse the above configuration file and save the configuration information of the configuration file for use in subsequent steps.

[0067] 2. Get the file list based on the file name to be loaded.

[0068] 3. Loop through the records in each file and check whether the record line length is consistent with that in the configuration file according to the configuration information.

[0069] 4. If the lengths are inconsistent, the log is registered and the record is skipped. If they are consistent, the record is trimmed according to the configuration information, trimmed into the content of each field, and the corresponding relationship is established with the fields in the configuration information.

[0070] 5. Process the field content according to the default value and processing function for each field in the configuration information.

[0071] 6. If the processing function is a custom built-in function, the processing is completed in the local Java program, saving database processing resources. If the processing function is not a custom built-in function, it is handled by the database when loading data.

[0072] 7. Assemble the processed field values ​​into different batch SQL scripts according to the loading database. For example, for MYSQL database, you can use the following script format. If the database record does not exist, insert it; if it exists, update the original record: INSERT INTO TABLENAME(A,B)VALUES('VALUEA','VALUEB'),('VALUEC','VALUED')ONDUPLICATE KEY UPDATE A=VALUES(A),B=VALUES(B).

[0073] 8. To ensure the use of small transactions, submit the database for loading every time 500 records are processed.

[0074] 9. Repeat this cycle to process the records of all files to be loaded.

[0075] Compared with the traditional database data loading solution, this device has the following advantages:

[0076] First, the transaction size of traditional databases is controlled by the application side, which is not strictly controlled, and it is easy to cause large transactions, which will affect the database. This device uses small transactions of 500 transactions to load data. In comparison, during data loading, the performance of the database is less affected, avoiding the impact of data loading on the online use of the database.

[0077] Second, this device supports custom built-in functions. Compared with the traditional database loading method, which uses the database's computing resources for loading, this device transfers the computing resources to the easily scalable local cloud server resources, greatly saving the database's shared resources and ensuring the stability of the database.

[0078] Third, the traditional database loading method may require the database client to be installed locally before the file can be loaded. This is somewhat contrary to the concept of distributed platforms, cloud servers, small image resources, and ready-to-use images. This device is introduced in the form of a Jar package, without the need to introduce an additional client, which is more convenient to use.

[0079] Fourth, in the traditional database loading method, each database system uses a loading solution, which is heterogeneous. This solution can more conveniently implement a set of solutions to connect multiple heterogeneous databases by parsing files and assembling SQL scripts, reducing the cost of data migration and data loading between different databases.

[0080] Figure 4 is a schematic diagram of a data processing device provided according to an embodiment of the present application, such as Figure 4 As shown, in order to achieve the above purpose, according to another aspect of the present application, a data processing device is provided, including: an acquisition module 42, a trimming module 44, a processing module 46 and a storage module 48, and the device is described in detail below.

[0081] An acquisition module 42 is used to acquire a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and the multiple table fields in the database table respectively correspond to the file positions of the file to be loaded; a trimming module 44 is connected to the acquisition module 42 and is used to trim the file to be loaded according to the configuration file and determine the multiple file fields corresponding to the multiple table fields; a processing module 46 is connected to the trimming module 44 and is used to process the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; a storage module 48 is connected to the processing module 46 and is used to store the processed file fields in the corresponding database.

[0082] Through the above device, the acquisition module 42 is used to acquire the file to be loaded and the configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of the database table, and the multiple table fields in the database table correspond to the file position of the file to be loaded respectively; the trimming module 44 trims the file to be loaded according to the configuration file, and determines the multiple file fields corresponding to the multiple table fields; the processing module 46 processes the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; the storage module 48 stores the processed file fields in the corresponding database. The relationship between the file to be loaded and the table fields of the database table is configured through the configuration file, and the file to be loaded is loaded into the database after being processed according to the processing functions corresponding to the table fields, which solves the problem of poor stability and low efficiency of the distributed system in the related art when loading large batches of data. Thus, the large batches of data files can be loaded smoothly and quickly, and the impact on the database performance is small, which improves the stability and loading efficiency of the distributed system in large batches of data loading.

[0083] The data processing device includes a processor and a memory. The acquisition module 42, the trimming module 44, the processing module 46 and the storage module 48 are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0084] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to detect whether the target image has a target object through the target object detection model, and if the target object has a target object, identify whether the posture of the target object is abnormal.

[0085] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data processing method is implemented.

[0087] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data processing method is executed when the program is run.

[0088] Figure 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application, such as Figure 5 As shown, an embodiment of the present invention provides an electronic device, the electronic device 50 includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program:

[0089] Obtain a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and multiple table fields in the database table respectively correspond to the file positions of the file to be loaded; trim the file to be loaded according to the configuration file, and determine multiple file fields corresponding to the multiple table fields; process the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; and store the processed file fields in the corresponding database.

[0090] Optionally, before the file to be loaded is trimmed according to the configuration file and multiple file fields corresponding to multiple table fields are determined, the method also includes: obtaining a file list of the files to be loaded, wherein the file list includes the file names of the multiple files to be loaded; parsing the configuration file to determine the file name of the file to be loaded and the database table name, wherein the multiple table fields in the database table respectively correspond to the file locations of the files to be loaded; traversing the file list and checking the multiple files to be loaded.

[0091] Optionally, traversing the file list and checking multiple files to be loaded include: traversing the file list and reading the length of the file line of the file to be loaded; determining the length of the file line corresponding to the table field of the database table according to the configuration file; when the length of the read file line is consistent with the length of the file line corresponding to the database table, executing the step of trimming the file to be loaded according to the configuration file and determining multiple file fields corresponding to the multiple table fields; when the length of the read file line is inconsistent with the length of the file line corresponding to the database table, skipping the file line of the file to be loaded, checking subsequent file lines, and logging the skipped file lines.

[0092] Optionally, the file name of the file to be loaded, and the database table name, and multiple table fields in the database table respectively correspond to the file position of the file to be loaded, including: according to the file position corresponding to the table field, the content data at the file position in the file to be loaded is cropped; when the cropped content data is not empty, the cropped content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed; when the cropped content data is empty, the preset default content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed.

[0093] Optionally, processing the file field corresponding to the table field according to the processing function corresponding to the table field includes: determining the processing function corresponding to the table field in the corresponding relationship according to the corresponding relationship, wherein the processing function corresponding to the table field is pre-set in the configuration file; and processing the file field corresponding to the corresponding relationship through the processing function.

[0094] Optionally, storing the processed file field in the corresponding database includes: when the processing function is a built-in function, processing the file field locally through the built-in function; when the processing function is a non-built-in function, sending the file field to the database, wherein the database processes the file field through the non-built-in function before loading the file field.

[0095] Optionally, storing the processed file fields in a corresponding database includes: determining a loading script corresponding to the database according to the processed file fields; and storing the file fields in the database by running the loading script.

[0096] Optionally, there are multiple databases, and before processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields, the method also includes: according to the identification field in the file to be loaded, wherein the identification field is used to identify the database corresponding to the file field in the file to be loaded; according to the identification field and the file field identified by the identification field, determining the database identifier stored in the file field.

[0097] Optionally, before storing the processed file fields in the corresponding database, the method also includes: recording the number of file lines that have been processed in the file to be loaded; when the number of file lines reaches a preset number, storing the processed file fields in the file to be loaded in the corresponding database.

[0098] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0099] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: obtaining a file to be loaded and a configuration file, wherein the configuration file includes a file name of the file to be loaded and a table name of a database table, and a plurality of table fields in the database table respectively correspond to the file positions of the file to be loaded; trimming the file to be loaded according to the configuration file, and determining a plurality of file fields corresponding to the plurality of table fields; processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields; and storing the processed file fields in the corresponding database.

[0100] Optionally, before the file to be loaded is trimmed according to the configuration file and multiple file fields corresponding to multiple table fields are determined, the method also includes: obtaining a file list of the files to be loaded, wherein the file list includes the file names of the multiple files to be loaded; parsing the configuration file to determine the file name of the file to be loaded and the database table name, wherein the multiple table fields in the database table respectively correspond to the file locations of the files to be loaded; traversing the file list and checking the multiple files to be loaded.

[0101] Optionally, traversing the file list and checking multiple files to be loaded include: traversing the file list and reading the length of the file line of the file to be loaded; determining the length of the file line corresponding to the table field of the database table according to the configuration file; when the length of the read file line is consistent with the length of the file line corresponding to the database table, executing the step of trimming the file to be loaded according to the configuration file and determining multiple file fields corresponding to the multiple table fields; when the length of the read file line is inconsistent with the length of the file line corresponding to the database table, skipping the file line of the file to be loaded, checking subsequent file lines, and logging the skipped file lines.

[0102] Optionally, the file name of the file to be loaded, and the database table name, and multiple table fields in the database table respectively correspond to the file position of the file to be loaded, including: according to the file position corresponding to the table field, the content data at the file position in the file to be loaded is cropped; when the cropped content data is not empty, the cropped content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed; when the cropped content data is empty, the preset default content data is used as the file field, a corresponding relationship is established with the table field, and the storage is performed.

[0103] Optionally, processing the file field corresponding to the table field according to the processing function corresponding to the table field includes: determining the processing function corresponding to the table field in the corresponding relationship according to the corresponding relationship, wherein the processing function corresponding to the table field is pre-set in the configuration file; and processing the file field corresponding to the corresponding relationship through the processing function.

[0104] Optionally, storing the processed file field in the corresponding database includes: when the processing function is a built-in function, processing the file field locally through the built-in function; when the processing function is a non-built-in function, sending the file field to the database, wherein the database processes the file field through the non-built-in function before loading the file field.

[0105] Optionally, storing the processed file fields in a corresponding database includes: determining a loading script corresponding to the database according to the processed file fields; and storing the file fields in the database by running the loading script.

[0106] Optionally, there are multiple databases, and before processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields, the method also includes: according to the identification field in the file to be loaded, wherein the identification field is used to identify the database corresponding to the file field in the file to be loaded; according to the identification field and the file field identified by the identification field, determining the database identifier stored in the file field.

[0107] Optionally, before storing the processed file fields in the corresponding database, the method also includes: recording the number of file lines that have been processed in the file to be loaded; when the number of file lines reaches a preset number, storing the processed file fields in the file to be loaded in the corresponding database.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0117] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtain a file to be loaded and a configuration file, wherein the configuration file includes the file name of the file to be loaded and the table name of a database table, and a plurality of table fields in the database table respectively correspond to the file locations of the file to be loaded; Tailoring the file to be loaded according to the configuration file to determine a plurality of file fields corresponding to the plurality of table fields; Processing the file field corresponding to the table field according to the processing function corresponding to the table field; Store the processed file fields into the corresponding database; Wherein, before the file to be loaded is cut according to the configuration file and a plurality of file fields corresponding to the plurality of table fields are determined, the method further comprises: Obtain a file list of the files to be loaded, wherein the file list includes file names of multiple files to be loaded; Parsing the configuration file to determine the file name of the file to be loaded and the name of the database table, wherein the multiple table fields in the database table respectively correspond to the file locations of the file to be loaded; Traversing the file list, and checking the plurality of files to be loaded; The file name of the file to be loaded, and the database table name, wherein the multiple table fields in the database table correspond to the file locations of the file to be loaded respectively and include: According to the file position corresponding to the table field, cutting the content data at the file position in the file to be loaded; In the case that the clipped content data is not empty, the clipped content data is used as the file field, a corresponding relationship is established with the table field, and the data is stored; In the case that the clipped content data is empty, the preset default content data is used as the file field, a corresponding relationship is established with the table field, and the data is stored.

2. The method according to claim 1, characterized in that Traversing the file list and checking the plurality of files to be loaded includes: Traverse the file list and read the length of the file line of the file to be loaded; Determine the length of the file line corresponding to the table field of the database table according to the configuration file; When the length of the read file line is consistent with the length of the file line corresponding to the database table, performing the step of trimming the file to be loaded according to the configuration file to determine a plurality of file fields corresponding to the plurality of table fields; When the length of the read file line is inconsistent with the length of the file line corresponding to the database table, the file line of the file to be loaded is skipped, the subsequent file lines are checked, and the skipped file lines are registered in the log.

3. The method according to claim 1, characterized in that According to the processing function corresponding to the table field, processing the file field corresponding to the table field includes: Determine, according to the corresponding relationship, a processing function corresponding to the table field in the corresponding relationship, wherein the processing function corresponding to the table field is pre-set in the configuration file; The file field corresponding to the corresponding relationship is processed by the processing function.

4. The method according to claim 3, characterized in that Storing the processed file fields in the corresponding database includes: In the case where the processing function is a built-in function, the file field is processed locally by the built-in function; In the case where the processing function is a non-built-in function, the file field is sent to the database, wherein the database processes the file field by means of the non-built-in function before loading the file field.

5. The method according to claim 1, characterized in that Storing the processed file fields in the corresponding database includes: Determining a loading script corresponding to the database according to the processed file fields; By running the loading script, the file fields are stored in the database.

6. The method according to claim 5, characterized in that There are multiple databases, and before processing the file fields corresponding to the table fields according to the processing functions corresponding to the table fields, the method further includes: According to the identification field in the file to be loaded, wherein the identification field is used to identify the database corresponding to the file field in the file to be loaded; According to the identification field and the file field identified by the identification field, a database identifier stored in the file field is determined.

7. The method according to any one of claims 1 to 6, characterized in that Before storing the processed file fields in the corresponding database, the method further includes: Record the number of file lines whose processing has been completed for the file to be loaded; When the number of the file lines reaches a preset number, the processed file fields in the file to be loaded are stored in the corresponding database.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire a file to be loaded and a configuration file, wherein the configuration file includes a file name of the file to be loaded and a table name of a database table, and a plurality of table fields in the database table respectively correspond to the file locations of the file to be loaded; A trimming module, used for trimming the file to be loaded according to the configuration file, and determining a plurality of file fields corresponding to the plurality of table fields; A processing module, used for processing the file field corresponding to the table field according to the processing function corresponding to the table field; The storage module is used to store the processed file fields in the corresponding database. The device is further used for cutting the file to be loaded according to the configuration file, and obtaining a file list of the file to be loaded before determining the multiple file fields corresponding to the multiple table fields, wherein the file list includes the file names of the multiple files to be loaded; parsing the configuration file to determine the file name of the file to be loaded and the database table name, wherein the multiple table fields in the database table respectively correspond to the file locations of the file to be loaded; traversing the file list to check the multiple files to be loaded; The device is also used to crop the content data at the file position in the file to be loaded according to the file position corresponding to the table field; when the cropped content data is not empty, the cropped content data is used as the file field, a corresponding relationship is established with the table field, and the data is stored; when the cropped content data is empty, the preset default content data is used as the file field, a corresponding relationship is established with the table field, and the data is stored.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method described in any one of claims 1 to 7.

Images