Data analysis and storage method and device and computer equipment

Abstract

The invention relates to a data analysis and storage method and device and computer equipment. The method comprises the steps that a mapping relation configuration file of original data is loaded; acquiring information of each data item in the original data according to the mapping relation configuration file; traversing each data item, and for the first data item with the corresponding database table and field, analyzing the first data item according to the data protocol type of the first data item; for the second data item without the corresponding database table and field, performing direct skipping processing on the second data item according to the data length of the second data item; generating an SQL statement according to an analyzed result; and executing the SQL statement to complete database storage updating. Through the data analysis and storage method and device, the problems that in the prior art, data analysis and storage need to be manually written by developers, and the development amount is large are solved, and the effects of reducing the development workload and improving the overall working efficiency of data analysis and storage are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a data parsing and warehousing method, device, and computer device. Background Art

[0002] An information system needs to collect or receive various raw data from various data sources, perform parsing and warehousing processing on the raw data, and finally provide it for users through visualization, statistical analysis, reports, or other means.

[0003] Data parsing plays a crucial role in an information system. It can help the system effectively process and utilize various data resources, improve data quality, system stability, and decision-making ability, and is a key link in the operation and development of the information system.

[0004] For data parsing and warehousing, an ORM framework is usually adopted, and the main solutions are as follows:

[0005] 1. Write the code for the data parsing module according to the raw data protocol, and usually parse all data items according to the protocol.

[0006] 2. Manually write the object entity class (Object) according to the database table structure, and usually it is a complete mapping of the database table.

[0007] 3. Since in actual business, it is usually not necessary to parse and warehouse all data items of the raw data, it is necessary to write the code for the screening and mapping module to assign the parsed raw data items required by the business to the corresponding fields of the object entity class.

[0008] As Figure 1 shown, the main processing flow of data parsing and warehousing in the related art is as follows:

[0009] 1. Parse the raw data according to the corresponding data protocol;

[0010] 2. Assign the parsed data to the ORM object;

[0011] 3. The ORM framework maps the ORM object to SQL to operate the database.

[0012] The data parsing and warehousing solution using the ORM framework in the related art has the following disadvantages:

[0013] 1. The raw data parsing module class needs to be manually written by developers, with a large development volume and redundancy;

[0014] 2. The ORM object entity class needs to be manually written by developers, with a large development volume and redundancy;

[0015] 3. The screening and mapping module needs to be manually written by developers, with a large development volume;

[0016] 4. Non-developers cannot participate in this work.

[0017] Currently, in the related art, developers need to manually write code for data parsing and storage into the database, and there is a large amount of development work. However, no effective solution has been proposed yet. Summary of the Invention

[0018] The purpose of this application is to address the deficiencies in the prior art and provide a data parsing and storage method, device, computer device, and computer-readable storage medium to at least solve the problem that developers need to manually write code for data parsing and storage into the database, resulting in a large amount of development work.

[0019] To achieve the above purpose, the technical solution adopted by this application is as follows:

[0020] In a first aspect, an embodiment of this application provides a data parsing and storage method, including:

[0021] Loading a mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables, and fields of each data item in the original data;

[0022] Obtaining information of each data item in the original data according to the mapping relationship configuration file;

[0023] Traversing each data item. For a first data item with a corresponding database table and field, parsing the first data item according to the data protocol type of the first data item; for a second data item without a corresponding database table and field, directly skipping the second data item according to the data length of the second data item;

[0024] Generating an SQL statement according to the parsed result;

[0025] Executing the SQL statement to complete database storage and update.

[0026] In some of these embodiments, the traversing each data item, for a first data item with a corresponding database table and field, parsing the first data item according to the data protocol type of the first data item; for a second data item without a corresponding database table and field, directly skipping the second data item according to the data length of the second data item includes:

[0027] Setting the initial value of the cursor to 0;

[0028] Traversing each data item, and performing the following steps for each data item until all data items are traversed:

[0029] Obtain the target data indicated by the data length from the original data;

[0030] Set the value of the cursor to increase by the data length;

[0031] Determine whether the corresponding database table and field are empty;

[0032] If the corresponding database table and field are not empty, call the parsing function according to the data protocol type to parse the target data;

[0033] Add the parsed data to the value array;

[0034] Add the corresponding database table and field to the key array.

[0035] In some embodiments, generating the SQL statement according to the parsed result includes:

[0036] Obtain the database table field list according to the value array;

[0037] Obtain the value list corresponding to the database table fields according to the key array;

[0038] Generate an SQL statement according to the database table field list and the value list.

[0039] In some embodiments, after determining whether the corresponding database table and field are empty, it further includes:

[0040] If the corresponding database table and field are empty, skip the target data from the original data.

[0041] In a second aspect, an embodiment of the present application provides a data parsing and warehousing device, including:

[0042] A loading unit, configured to load a mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data;

[0043] An obtaining unit, configured to obtain the information of each data item in the original data according to the mapping relationship configuration file;

[0044] A parsing unit, configured to traverse each data item. For the first data item with a corresponding database table and field, parse the first data item according to the data protocol type of the first data item; for the second data item without a corresponding database table and field, directly skip the second data item according to the data length of the second data item;

[0045] A generation unit, used to generate SQL statements according to the parsed results;

[0046] The storage unit is used to execute the SQL statement to complete the database storage update.

[0047] In some embodiments, the parsing unit includes:

[0048] The setting module is used to set the initial value of the cursor to 0;

[0049] The traversal module is used to traverse the data items and perform the following steps on each data item until all data items are traversed:

[0050] Acquire target data indicated by the data length from the original data;

[0051] Setting the value of the cursor increases the data length;

[0052] Determine whether the corresponding database table and field are empty;

[0053] If the corresponding database table and field are not empty, calling the parsing function according to the data protocol type to parse the target data;

[0054] Add the parsed data to the value array;

[0055] Add the corresponding database table and field to the key array.

[0056] In some embodiments, the generating unit comprises:

[0057] A first acquisition module, used for acquiring a database table field list according to the value array;

[0058] A second acquisition module, used to acquire a value list corresponding to a database table field according to the key array;

[0059] A generating module is used to generate an SQL statement according to the database table field list and the value list.

[0060] In some embodiments, the parsing unit further comprises:

[0061] The skip module is used to skip the target data from the original data if the corresponding database table and field are empty.

[0062] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data parsing and warehousing method as described in the first aspect above is implemented.

[0063] Fourthly, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the data parsing and warehousing method as described in the first aspect above.

[0064] The present application adopts the above technical solutions. Compared with the prior art, the data parsing and warehousing method provided by the embodiment of the present application loads a mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data; obtains information of each data item in the original data according to the mapping relationship configuration file; traverses each data item, and for the first data item with a corresponding database table and field, parses the first data item according to the data protocol type of the first data item; for the second data item without a corresponding database table and field, directly skips the second data item according to the data length of the second data item; generates an SQL statement according to the parsed result; executes the SQL statement to complete database warehousing and update, solves the problem that developers need to manually write code for data parsing and warehousing in the related art, and has a large development workload, and achieves the effects of reducing the development workload and improving the overall work efficiency of data parsing and warehousing.

[0065] Details of one or more embodiments of the present application are set forth in the following drawings and description, so that other features, objects, and advantages of the present application will become more comprehensible. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0067] Figure 1 is a schematic flow chart of data parsing and warehousing according to the related art;

[0068] Figure 2 is a block diagram of a mobile terminal according to an embodiment of the present application;

[0069] Figure 3 is a flow chart of a data parsing and warehousing method according to an embodiment of the present application;

[0070] Figure 4 is a schematic flow chart of a low-code data parsing and warehousing method according to a preferred embodiment of the present application;

[0071] Figure 5 is a schematic diagram of the execution process of low-code data parsing and warehousing according to a preferred embodiment of the present application;

[0072] Figure 6It is a structural block diagram of a data parsing and warehousing device according to an embodiment of the present application;

[0073] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. Detailed implementation manners

[0074] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be described and explained below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided in the present application without creative efforts fall within the scope of protection of the present application.

[0075] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar scenarios based on these drawings without creative efforts. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be understood as the content disclosed in the present application being insufficient.

[0076] Referring to "embodiments" in the present application means that specific features, structures or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0077] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those with ordinary skills in the technical field to which this application pertains. The words such as "a", "an", "one", "the" and the like involved in this application do not indicate a limitation in quantity and may represent a singular or plural number. The terms "include", "comprise", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may further include unlisted steps or units, or may further include other steps or units inherent to these processes, methods, products or devices. The words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The terms "first", "second", "third" and the like involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0078] First, the following technical terms in the embodiments of this application are explained as follows:

[0079] AFC: The AFC system generally refers to the "Automatic Fare Collection System", that is, the automatic fare collection and inspection system.

[0080] ORM: It refers to Object-Relational Mapping, which is a technology for mapping data between an object model and a relational database. The ORM framework can help developers operate the database during programming without directly writing SQL statements, simplifying the process of database operation.

[0081] This embodiment provides a mobile terminal. Figure 2 It is a structural block diagram of the mobile terminal according to the embodiments of this application. As Figure 2 shown, the mobile terminal includes components such as a Radio Frequency (RF) circuit 210, a memory 220, an input unit 230, a display unit 240, a sensor 250, an audio circuit 260, a wireless fidelity (WiFi) module 270, a processor 280, and a power supply 290. Those skilled in the art can understand, Figure 2The mobile terminal structure shown does not limit the mobile terminal, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0082] The following will specifically introduce each component of the mobile terminal in combination with Figure 2 :

[0083] The RF circuit 210 can be used for receiving and sending information or signals during a call. Specifically, after receiving the downlink information from the base station, it is given to the processor 280 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to antennas, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (abbreviated as LNA), a duplexer, etc. In addition, the RF circuit 210 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (abbreviated as GSM), General Packet Radio Service (abbreviated as GPRS), Code Division Multiple Access (abbreviated as CDMA), Wideband Code Division Multiple Access (abbreviated as WCDMA), Long Term Evolution (abbreviated as LTE), email, Short Messaging Service (abbreviated as SMS), etc.

[0084] The memory 220 can be used to store software programs and modules. The processor 280 executes various functional applications and data processing of the mobile terminal by running the software programs and modules stored in the memory 220. The memory 220 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, applications required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile terminal (such as audio data, a phone book, etc.). In addition, the memory 220 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0085] The input unit 230 can be used to receive input numerical or character information, and generate key signal inputs related to the user settings and function controls of the mobile terminal. Specifically, the input unit 230 can include a touch panel 231 and other input devices 232. The touch panel 231, also known as a touch screen, can collect touch operations of the user thereon or nearby (such as operations of the user using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch panel 231), and drive corresponding connection devices according to a preset program. Optionally, the touch panel 231 can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch orientation of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, then sends it to the processor 280, and can receive and execute commands sent by the processor 280. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch panel 231. In addition to the touch panel 231, the input unit 230 can also include other input devices 231. Specifically, the other input devices 231 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0086] The display unit 240 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile terminal. The display unit 240 can include a display panel 241. Optionally, the display panel 241 can be configured in forms such as a liquid crystal display (LCD for short) and an organic light-emitting diode (OLED for short). Further, the touch panel 231 can cover the display panel 241. When the touch panel 231 detects a touch operation thereon or nearby, it transmits it to the processor 280 to determine the type of touch event. Subsequently, the processor 280 provides corresponding visual output on the display panel 241 according to the type of touch event. Although in Figure 2 the touch panel 231 and the display panel 241 are implemented as two independent components to realize the input and input functions of the mobile terminal, in some embodiments, the touch panel 231 and the display panel 241 can be integrated to realize the input and output functions of the mobile terminal.

[0087] The mobile terminal may further include at least one sensor 250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display panel 241 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 241 and / or the backlight when the mobile terminal is moved to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used in applications for identifying the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile terminal can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be elaborated here.

[0088] The speaker 261 and the microphone 262 in the audio circuit 260 can provide an audio interface between the user and the mobile terminal. The audio circuit 260 can transmit the electrical signal converted from the received audio data to the speaker 261, and the speaker 261 converts it into a sound signal for output; on the other hand, the microphone 262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 260 and then converted into audio data. After the audio data is output to the processor 280 for processing, it is sent through the RF circuit 210 to, for example, another mobile terminal, or the audio data is output to the memory 220 for further processing.

[0089] WiFi belongs to short - range wireless transmission technology. The mobile terminal can help users send and receive emails, browse the web, and access streaming media through the WiFi module 270, which provides users with wireless broadband Internet access. Although Figure 2 the WiFi module 270 is shown, it can be understood that it does not belong to an essential component of the mobile terminal and can be completely omitted or replaced with other short - range wireless transmission modules, such as Zigbee modules, or WAPI modules, etc., within the scope of not changing the essence of the invention as needed.

[0090] The processor 280 is the control center of the mobile terminal, connecting various parts of the entire mobile terminal through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 220, and by calling data stored in the memory 220, it executes various functions of the mobile terminal and processes data, thereby monitoring the mobile terminal as a whole. Optionally, the processor 280 may include one or more processing units; preferably, the processor 280 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 280 either.

[0091] The mobile terminal further includes a power supply 290 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 280 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system.

[0092] Although not shown, the mobile terminal may further include a camera, a Bluetooth module, etc., which will not be elaborated here.

[0093] In this embodiment, the processor 280 is configured to:

[0094] Load a mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data;

[0095] Obtain information of each data item in the original data according to the mapping relationship configuration file;

[0096] Traverse each data item. For a first data item with a corresponding database table and field, parse the first data item according to the data protocol type of the first data item; for a second data item without a corresponding database table and field, directly skip the second data item according to the data length of the second data item;

[0097] Generate an SQL statement according to the parsed result;

[0098] Execute the SQL statement to complete the database storage update.

[0099] In some of the embodiments, the processor 280 is further configured to:

[0100] Set the initial value of the cursor to 0;

[0101] Traverse each data item, and perform the following steps for each data item until all data items are traversed:

[0102] Obtain the target data indicated by the data length from the original data;

[0103] Set the value of the cursor to increase by the data length;

[0104] Judge whether the corresponding database table and field are empty;

[0105] If the corresponding database table and field are not empty, call a parsing function according to the data protocol type to parse the target data;

[0106] Add the parsed data to the value array;

[0107] Add the corresponding database table and fields to the key array.

[0108] In some embodiments, the processor 280 is further configured to:

[0109] Obtain a list of database table fields according to the value array;

[0110] Obtain a list of values corresponding to the database table fields according to the key array;

[0111] Generate an SQL statement according to the database table field list and the value list.

[0112] In some embodiments, the processor 280 is further configured to:

[0113] If the corresponding database table and fields are empty, skip the target data from the original data.

[0114] This embodiment provides a method for parsing and storing data into a database. Figure 3 It is a flowchart of the method for parsing and storing data into a database according to an embodiment of the present application, as Figure 3 shown, and the process includes the following steps:

[0115] Step S301, load the mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data;

[0116] Step S302, obtain the information of each data item in the original data according to the mapping relationship configuration file;

[0117] Step S303, traverse each data item. For the first data item with corresponding database table and fields, parse the first data item according to the data protocol type of the first data item; for the second data item without corresponding database table and fields, directly skip the second data item according to the data length of the second data item;

[0118] Step S304, generate an SQL statement according to the parsed result;

[0119] Step S305, execute the SQL statement to complete the database storage and update.

[0120] In the above step S301, the mapping relationship configuration file can adopt a document form that is convenient to edit, easy to understand, and has high readability, such as Excel. The mapping relationship configuration file of the original data can be used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data. A specific example is shown in Table 1 below:

[0121] Table 1

[0122]

[0123] According to the above mapping relationship configuration file, information about each data item in the original data can be obtained. In the embodiment of the present application, step S302 can define the following data structure:

[0124]

[0125] Then, according to the mapping relationship configuration file, the following Node array can be obtained:

[0126]

[0127] The above Node array can be used to describe the information of each data item in the original data.

[0128] In the embodiment of the present application, through the above step S303, each of the data items is traversed. For the first data item with a corresponding database table and fields, the first data item is parsed according to the data protocol type of the first data item; for the second data item without a corresponding database table and fields, the second data item is directly skipped according to the data length of the second data item, so as to achieve the purpose of parsing the data items in the original data, without the need to compile a large amount of code. By using the mapping relationship configuration file, data parsing can be achieved, realizing the effect of reducing the development workload and improving the data parsing efficiency.

[0129] In some of the embodiments, in step S303, each of the data items is traversed. For the first data item with a corresponding database table and fields, the first data item is parsed according to the data protocol type of the first data item; for the second data item without a corresponding database table and fields, the second data item is directly skipped according to the data length of the second data item, including:

[0130] Set the initial value of the cursor to 0;

[0131] Traverse each of the data items, and perform the following steps for each data item until all data items are traversed:

[0132] Obtain the target data indicated by the data length from the original data;

[0133] Set the value of the cursor to increase by the data length;

[0134] Judge whether the corresponding database table and fields are empty;

[0135] If the corresponding database table and fields are not empty, call the parsing function according to the data protocol type to parse the target data;

[0136] Add the parsed data to the value array;

[0137] Add the corresponding database table and fields to the key array.

[0138] Optionally, if the corresponding database table and fields are empty, skip the target data from the original data.

[0139] It should be noted that the embodiments of the present application can provide the parsing function of the basic data types in the protocol according to the protocol of the original data, and only support the parsing process of the basic elements (i.e., data items) in the original data, without providing the parsing process of the complete data. Write the corresponding parsing functions or methods according to the basic types specified in the original data protocol to be parsed, such as Char, uint8, uint16, uint32 or custom types, etc.

[0140] Examples of the protocol basic data parsing method are as follows:

[0141] Decode_CHAR(): Parse the Char data.

[0142] Brief description: Copy the data of the corresponding bytes from the original data to the character array.

[0143] Decode_Uint32(): Parse the Uint32 data.

[0144] Brief description: Copy 4 bytes from the original data, perform byte order conversion and assign them to a Uint32 variable.

[0145] According to the above optional embodiments, the parsing process for the above Node array is as follows:

[0146] 1. Process: {"Transaction version number", CHAR, 9, ""}

[0147] Original data RAW[0,9], skip without parsing.

[0148] Cursor = 9

[0149] Keys[] = {}

[0150] Values[] = {}

[0151] 2. Process: {"Transaction type", CHAR, 2, "TRAN_TYPE"}

[0152] Original data RAW[9,11], the parsed data is assumed to be 01.

[0153] Cursor = 11

[0154] Keys[] = {"TRAN_TYPE"}

[0155] Values[] = {"01"}

[0156] 3. Processing: {"Transaction Location", CHAR, 9, ""},

[0157] Original data RAW[11,20], skip without parsing.

[0158] Cursor = 20

[0159] Keys[] = {}

[0160] Values[] = {}

[0161] 4. Processing: {"Operator ID", "CHAR", 6, "OPERATOR_ID"}

[0162] Original data RAW[20,26], the parsed data is assumed to be 123456.

[0163] Cursor = 26

[0164] Keys[] = {"TRAN_TYPE", "OPERATOR_ID"}

[0165] Values[] = {"01", "123456"}

[0166] 5. Processing: {"Transaction Serial Number", "Uint32", 4, "UDSN"}

[0167] Original data RAW[26,30], the parsed data is assumed to be 1005.

[0168] Cursor = 30

[0169] Keys[] = {"TRAN_TYPE", "OPERATOR_ID", "UDSN"}

[0170] Values[] = {"01", "123456", 1005}

[0171] 6. Processing: {"Transaction Amount", "Uint32", 4, "TRAN_VALUE"}

[0172] Original data RAW[30,34], the parsed data is assumed to be 100.

[0173] Cursor = 34

[0174] Keys[] = {"TRAN_TYPE", "OPERATOR_ID", "UDSN", "TRAN_VALUE"}

[0175] Values[] = {"01", "123456", 1005, 100}

[0176] After the above parsing, the following can be obtained:

[0177] The key array Keys[] = {"TRAN_TYPE", "OPERATOR_ID", "UDSN", "TRAN_VALUE"}

[0178] The value array Values[] = {"01", "123456", 1005, 100}

[0179] In the embodiments of the present application, by loading the mapping relationship configuration file, only the parts with mapping relationships are parsed and processed, and the parts without mapping relationships are skipped without parsing. According to the type of the original data item, the corresponding method in the basic parsing module is used for parsing, and finally the corresponding SQL statement is generated. This method can improve the data parsing efficiency and reduce the development workload.

[0180] In some of the embodiments, the step S304 of generating an SQL statement according to the parsed result may include:

[0181] Obtain the database table field list according to the value array;

[0182] Obtain the value list corresponding to the database table fields according to the key array;

[0183] Generate an SQL statement according to the database table field list and the value list.

[0184] Through step S304, an SQL statement can be generated according to the parsed result. The specific SQL statement is as follows:

[0185] Insert Into Transaction(TRAN_TYPE,OPERATOR_ID,UDSN,TRAN_VALUE)Values('01', '123456', 1005, 100)

[0186] Then execute the SQL statement to complete the database storage and update.

[0187] The embodiments of the present application will be described and illustrated below through preferred embodiments.

[0188] The low-code data parsing and warehousing method provided by this preferred embodiment can be used to solve problems such as the need to write a large amount of code for original data protocol parsing, screening and assignment, ORM object entity class definition, etc. during the data parsing and warehousing process, which is error-prone and inaccessible to non-developers. This method includes: Based on the low-code concept, a mapping relationship configuration file that is easy to edit, understand, and has high readability is used to describe the mapping relationship between the original data to be parsed, its data items, and database tables and fields. By loading the mapping relationship configuration file, the processing of parsing, screening and assignment, and SQL generation can be implemented through a low-code logical processing method, achieving the purpose of reducing development workload and improving data parsing efficiency.

[0189] Figure 4 is a schematic flowchart of the low-code data parsing and warehousing method according to the preferred embodiment of the present application, as Figure 4 shown, the parsing and mapping module can implement the on-demand parsing process. By loading the mapping relationship configuration file, it is possible to parse the data items in the original data that have a mapping relationship with the database table to obtain parsed data, and then generate an SQL statement according to the parsed data, and implement the database mapping and warehousing operation by executing the SQL statement.

[0190] Figure 5 is a schematic diagram of the execution process of the low-code data parsing and warehousing according to the preferred embodiment of the present application, as Figure 5 shown, this data parsing and warehousing process includes: loading the mapping relationship configuration file; determining whether there are unprocessed data items; if there are unprocessed data items, obtaining the data item information, and then determining whether there is a mapping relationship for this data item; if there is no mapping relationship for this data item, return to the step of determining whether there are unprocessed data items; if there is a mapping relationship for this data item, execute the parsed data item, and then return to the step of determining whether there are unprocessed data items; if there are no unprocessed data items, generate an SQL statement; update the database by executing the SQL statement.

[0191] The embodiment of the present application provides a low-code data parsing and warehousing implementation solution, which separates the business logic from the code, uses a document form that is easy to edit, understand, and has high readability to describe the business relationship, simplifies the parsing process and can perform on-demand parsing, eliminates redundant processing, combines the processing of parsing, screening and assignment, and SQL generation, reduces the development workload, and enables the business mapping relationship to be processed by non-developers familiar with the business, reducing the R & D resource requirements and improving the overall work efficiency.

[0192] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0193] This embodiment provides a data parsing and warehousing device, which is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, terms such as "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0194] Figure 6 is a structural block diagram of the data parsing and warehousing device according to an embodiment of the present application, as Figure 6 shown, the device includes:

[0195] A loading unit 61, configured to load a mapping relationship configuration file of the original data, where the mapping relationship configuration file is used to describe the names, data protocol types, data lengths, corresponding database tables and fields of each data item in the original data;

[0196] An obtaining unit 62, configured to obtain information of each data item in the original data according to the mapping relationship configuration file;

[0197] A parsing unit 63, configured to traverse each data item. For a first data item with a corresponding database table and field, parse the first data item according to the data protocol type of the first data item; for a second data item without a corresponding database table and field, directly skip the second data item according to the data length of the second data item;

[0198] A generating unit 64, configured to generate an SQL statement according to the parsed result;

[0199] A warehousing unit 65, configured to execute the SQL statement to complete database warehousing and update.

[0200] In some of these embodiments, the parsing unit 63 includes:

[0201] A setting module, configured to set the initial value of the cursor to 0;

[0202] A traversing module, configured to traverse each data item, and execute the following steps for each data item until all data items are traversed:

[0203] Obtain the target data indicated by the data length from the original data;

[0204] Set the value of the cursor to increase the data length;

[0205] Determine whether the corresponding database table and field are empty;

[0206] If the corresponding database table and field are not empty, call the parsing function according to the data protocol type to parse the target data;

[0207] Add the parsed data to the value array;

[0208] Add the corresponding database table and field to the key array.

[0209] In some embodiments, the generating unit 64 includes:

[0210] A first obtaining module, configured to obtain a database table field list according to the value array;

[0211] A second obtaining module, configured to obtain a value list corresponding to the database table field according to the key array;

[0212] A generating module, configured to generate an SQL statement according to the database table field list and the value list.

[0213] In some embodiments, the parsing unit 63 further includes:

[0214] A skipping module, configured to skip the target data from the original data if the corresponding database table and field are empty.

[0215] It should be noted that the above-mentioned each module can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, the above-mentioned each module can be located in the same processor; or the above-mentioned each module can also be located in different processors in any combined form.

[0216] The embodiment provides a computer device. The data parsing and storage method according to the embodiment of the present application can be implemented by the computer device. Figure 7 It is a schematic hardware structure diagram of a computer device according to an embodiment of the present application.

[0217] The computer device may include a processor 71 and a memory 72 storing computer program instructions.

[0218] Specifically, the above-mentioned processor 71 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present application.

[0219] Among them, the memory 72 may include a mass storage for data or instructions. By way of example and not limitation, the memory 72 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 72 may include removable or non-removable (or fixed) media. Where appropriate, the memory 72 may be internal or external to the data processing device. In a particular embodiment, the memory 72 is non-volatile memory. In a particular embodiment, the memory 72 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. Where appropriate, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended data out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0220] The memory 72 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 71.

[0221] The processor 71 reads and executes the computer program instructions stored in the memory 72 to implement any one of the data parsing and warehousing methods in the above embodiments.

[0222] In some of the embodiments, the computer device may further include a communication interface 73 and a bus 70. Among them, as Figure 7 shown, the processor 71, the memory 72, and the communication interface 73 are connected through the bus 70 to complete communication with each other.

[0223] The communication interface 73 is used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication interface 73 can also implement data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0224] The bus 70 includes hardware, software, or both, and couples components of a computer device to each other. The bus 70 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, Local Bus. By way of example and not limitation, the bus 70 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable bus or a combination of two or more of these. In a suitable case, the bus 70 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0225] In addition, in combination with the data parsing and warehousing method in the above embodiments, the embodiments of the present application can be implemented by providing a computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, any one of the data parsing and warehousing methods in the above embodiments is implemented.

[0226] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0227] The embodiments described above merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A data analysis and warehousing method, characterized in that: include: Loading a mapping relationship configuration file of the original data, wherein the mapping relationship configuration file is used to describe the name, data protocol type, data length, and corresponding database table and field of each data item in the original data; Acquire information of each data item in the original data according to the mapping relationship configuration file; Traversing the data items, for a first data item that has a corresponding database table and field, parsing the first data item according to the data protocol type of the first data item; for a second data item that does not have a corresponding database table and field, directly skipping the second data item according to the data length of the second data item; Generate SQL statements based on the parsed results; Execute the SQL statement to complete the database update.

2. The method according to claim 1, characterized in that The traversing of the data items, for a first data item that has a corresponding database table and field, parsing the first data item according to the data protocol type of the first data item; for a second data item that does not have a corresponding database table and field, directly skipping the second data item according to the data length of the second data item, includes: Set the initial value of the cursor to 0; Traverse the data items and perform the following steps on each data item until all data items are traversed: Acquire target data indicated by the data length from the original data; Setting the value of the cursor increases the data length; Determine whether the corresponding database table and field are empty; If the corresponding database table and field are not empty, calling the parsing function according to the data protocol type to parse the target data; Add the parsed data to the value array; Add the corresponding database table and field to the key array.

3. The method according to claim 2, characterized in that The generating of SQL statements according to the parsed results includes: Obtain a database table field list according to the value array; Obtain a value list corresponding to the database table field according to the key array; Generate an SQL statement according to the database table field list and the value list.

4. The method according to claim 2, characterized in that: After determining whether the corresponding database table and field are empty, the method further includes: If the corresponding database table and field are empty, the target data is skipped from the original data.

5. A data analysis and storage device, characterized in that: include: A loading unit, used to load a mapping relationship configuration file of the original data, wherein the mapping relationship configuration file is used to describe the name, data protocol type, data length, and corresponding database table and field of each data item in the original data; An acquiring unit, configured to acquire information of each data item in the original data according to the mapping relationship configuration file; a parsing unit, configured to traverse the data items, and for a first data item for which a corresponding database table and field exist, parse the first data item according to a data protocol type of the first data item; and for a second data item for which a corresponding database table and field do not exist, directly skip the second data item according to a data length of the second data item; A generation unit, used to generate SQL statements according to the parsed results; The storage unit is used to execute the SQL statement to complete the database storage update.

6. The device according to claim 5, characterized in that The parsing unit comprises: The setting module is used to set the initial value of the cursor to 0; The traversal module is used to traverse the data items and perform the following steps on each data item until all data items are traversed: Acquire target data indicated by the data length from the original data; Setting the value of the cursor increases the data length; Determine whether the corresponding database table and field are empty; If the corresponding database table and field are not empty, calling the parsing function according to the data protocol type to parse the target data; Add the parsed data to the value array; Add the corresponding database table and field to the key array.

7. The device according to claim 6, characterized in that The generating unit comprises: A first acquisition module, used for acquiring a database table field list according to the value array; A second acquisition module, used to acquire a value list corresponding to a database table field according to the key array; A generating module is used to generate an SQL statement according to the database table field list and the value list.

8. The device according to claim 7, characterized in that The parsing unit also includes: The skip module is used to skip the target data from the original data if the corresponding database table and field are empty.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images