Encoding data processing method and device, apparatus, and storage medium

Abstract

The application relates to the field of data processing, and discloses a coded data processing method, device, equipment and storage medium, the method comprises the following steps: reading a to-be-converted file according to a field definition table to obtain field-level data; decoding the field-level data through an encoding character set to obtain decoded data; obtaining a standardization processing function, and performing data cleaning on the decoded data according to the standardization processing function to obtain target data; and generating a general storage file based on the target data. Since the field-level data of the to-be-converted file is decoded through the encoding character set to obtain the decoded data, the decoded data is cleaned through the standardization processing function, the general storage file is generated based on the obtained target data, the data reading conversion of the file is realized, and the file data garbled code problem is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for coded data processing. Background Technology

[0002] Currently, IBM commercial computer products, such as the AS / 400, S390, and Power series, are widely used in industries such as finance and securities. Their data storage encoding largely adopts Extended Binary Coded Decimal Interchange Code (EBCDIC). These IBM commercial computers use EBCDIC as their storage encoding to store large amounts of business data. In order to use and mine this business data, big data scenarios require loading this data into a data lake.

[0003] However, commonly used data processing tools on the market, such as the open-source Kettle, do not support the processing of EBCDIC encoded data. When receiving this type of file from an IBM commercial computer, data garbled characters are likely to occur, making it impossible to process the received file normally.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is related technology. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for processing encoded data, aiming to solve the technical problem that existing data processing tools are prone to data garbled characters when receiving EBCDIC encoded files.

[0006] To achieve the above objectives, the present invention provides a method for processing encoded data, the method comprising the following steps:

[0007] Read the file to be converted from the field definition table to obtain field-level data;

[0008] The field-level data is decoded using an encoded character set to obtain the decoded data;

[0009] Obtain a standardized processing function, and perform data cleaning on the decoded data according to the standardized processing function to obtain the target data;

[0010] Generate a universal storage file based on the target data.

[0011] Optionally, the step of reading the file to be converted from the field definition table to obtain field-level data includes:

[0012] Obtain field definition information, which includes field width, field type, and field format;

[0013] The file to be converted is read based on a preset byte length, and the numerical information corresponding to each field in the file to be converted is obtained according to the field definition information to determine the field-level data.

[0014] Optionally, the method is applied to the Kettle system, and obtaining field definition information includes:

[0015] Import the COBOL file into the system and parse the COBOL file to obtain field definition information;

[0016] Alternatively, it can receive user commands, read the configured field list data in the system, and obtain field definition information.

[0017] Optionally, the step of decoding the field-level data using an encoded character set to obtain decoded data includes:

[0018] The field-level data is split into columns, and a decoding object corresponding to each column of field-level data is generated according to the encoded character set;

[0019] Obtain the numerical values ​​of each column's field-level data, and convert the numerical values ​​of each column's field-level data into binary values;

[0020] The decoded data is obtained by parsing the binary value using the decoding object.

[0021] Optionally, after parsing the binary value through the decoding object to obtain the decoded data, the process includes:

[0022] Obtain the field definition information and combine it with the decoded data to generate a mapping result set;

[0023] Accordingly, the step of obtaining a standardized processing function and performing data cleaning on the decoded data according to the standardized processing function to obtain target data includes:

[0024] Obtain a standardized processing function, and perform data cleaning on the decoded data in the mapping result set according to the standardized processing function to obtain the target data.

[0025] Optionally, obtaining the standardization processing function and cleaning the decoded data in the mapping result set according to the standardization processing function to obtain the target data includes:

[0026] The decoded data is filtered at the field level according to the preset row filtering conditions to obtain the filtered decoded data.

[0027] The filtered and decoded data is then processed in a format, including field renaming, field type conversion, and field order reordering.

[0028] The decoded data, after being formatted, is cleaned using a standardized processing function to obtain the target data.

[0029] Optionally, the general storage file includes: a data file, a control file, and a metadata file;

[0030] After generating the universal storage file based on the target data, the process includes:

[0031] Obtain the target storage database for the file to be converted;

[0032] The target storage database is filtered in the general storage file to obtain the current storage file corresponding to the target storage database.

[0033] Furthermore, to achieve the above objectives, the present invention also proposes an encoded data processing apparatus, the encoded data processing apparatus comprising:

[0034] The file reading module is used to read the file to be converted based on the field definition table and obtain field-level data.

[0035] The data decoding module is used to decode the field-level data using an encoded character set to obtain the decoded data;

[0036] The conversion and cleaning module is used to obtain a standardized processing function and perform data cleaning on the decoded data according to the standardized processing function to obtain the target data;

[0037] The data output module is used to generate a general storage file based on the target data.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes an encoded data processing device, which includes a memory, a processor, and an encoded data processing program stored in the memory and executable on the processor, the encoded data processing program being configured to implement the encoded data processing method described above.

[0039] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing an encoded data processing program, which, when executed by a processor, implements the encoded data processing method as described above.

[0040] This invention reads the file to be converted from a field definition table to obtain field-level data; decodes the field-level data using an encoded character set to obtain decoded data; obtains a standardization processing function and cleans the decoded data using the standardization processing function to obtain target data; and generates a universal storage file based on the target data. Because this invention decodes the field-level data of the file to be converted using an encoded character set to obtain decoded data, and then cleans the decoded data using a standardization processing function, thereby generating a universal storage file based on the obtained target data, it achieves data reading and conversion of files and solves the problem of garbled text in files. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of the encoding data processing device in the hardware operating environment involved in the embodiments of the present invention;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the encoded data processing method of the present invention;

[0043] Figure 3 This is a flowchart illustrating the second embodiment of the encoded data processing method of the present invention;

[0044] Figure 4 This is a flowchart illustrating the third embodiment of the encoded data processing method of the present invention;

[0045] Figure 5 This is a matching diagram between a general storage file and a target storage database in the third embodiment of the encoded data processing method of the present invention;

[0046] Figure 6 This is a structural block diagram of the first embodiment of the encoded data processing device of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the encoding data processing device in the hardware operating environment involved in the embodiments of the present invention.

[0050] like Figure 1As shown, the encoded data processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the coded data processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0052] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an encoded data processing program.

[0053] exist Figure 1 In the encoded data processing device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the encoded data processing device of the present invention can be set in the encoded data processing device, and the encoded data processing device calls the encoded data processing program stored in the memory 1005 through the processor 1001 and executes the encoded data processing method provided in the embodiment of the present invention.

[0054] This invention provides a method for processing encoded data, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the encoded data processing method of the present invention.

[0055] In this embodiment, the encoded data processing method includes the following steps:

[0056] Step S10: Read the file to be converted according to the field definition table to obtain field-level data.

[0057] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer. It can also be other electronic devices that can perform the same or similar functions and implement the encoded data processing method. This embodiment does not limit the scope of the invention. Here, an encoded data processing device (hereinafter referred to as the processing device) is selected as an example to illustrate various embodiments of the encoded data processing method of the invention.

[0058] It should be noted that this embodiment can be applied to scenarios requiring the processing of Extended Binary Coded Decimal Interchange Code (EBCDIC) format data. For example, it can be used to convert received file data from IBM AS / 400 and S390 commercial computers into common encoding formats such as GBK and UTF so that downstream applications can use the file data. AS / 400 and S390 are commonly used all-in-one IBM computers; EBCDIC is a binary encoding of alphanumeric characters, used by the IBM computer's operating system. EBDIC differs from the American Standard Code for Information Interchange (ASCII) and uses an 8-bit encoding set.

[0059] It should be noted that Kettle is typically used to process file data in data processing scenarios. Kettle is an open-source ETL tool used to perform data extraction, cleaning, transformation, and loading processes. Standard Kettle does not support EBCDIC format data processing, but EBCDIC file reading plugins can be developed based on Kettle's plugin extension mechanism. The encoded data processing method in this embodiment can be implemented based on this Kettle plugin.

[0060] It is understandable that the field definition table can be a table that stores field definition information, which contains pre-set field-level definition rules for reading files of different formats.

[0061] It is understood that the file to be converted can be an EBCDIC format file. This file can be a file received directly from an IBM commercial computer to the current processing device, or it can be an EBCDIC format file received from an upstream device in the data chain of the current processing device that requires data encoding conversion.

[0062] It should be understood that by reading the content of the file to be converted based on the field definition table, the field content of the file to be converted can be split based on the field definition to obtain field-level data.

[0063] Furthermore, to improve the reading efficiency of the file to be converted, step S10 includes:

[0064] Step S101: Obtain field definition information, which includes field width, field type, and field format.

[0065] It should be noted that the field definition information is stored in the field definition table mentioned above. The field width can be the definition of the unit field data to be read in the file to be read or the storage space occupied by each row of bytes of data; the field type can be the definition of different types of fields in the file to be read, such as integer fields, floating-point fields and fixed-point fields; the field format can be used to determine the reading rules for special format fields, such as date format fields.

[0066] Furthermore, considering that the method of this embodiment is applied to the Kettle system, and Kettle is an open-source tool, a field definition table containing field definition information can be obtained from multiple sources. Step S101 includes:

[0067] Step S1011: Import the COBOL file into the system and parse the COBOL file to obtain field definition information.

[0068] It should be noted that Common Business-Oriented Language (COBOL) is a widely used high-level programming language capable of handling enterprise management data that is relatively simple in numerical calculations but involves large volumes of data. COBOL files can be imported through Kettle's interactive interface, and then parsed to obtain field definitions, read file content, and split the content of the file to be converted into sub-segments according to the field definition table stored in the COBOL file.

[0069] In the specific implementation, the processing device parses the imported Kettle COBOL file and automatically maps the parsed COBOL content to Kettle's field definitions. These field definitions can include the width, type, format of each byte, and the width of each line of bytes.

[0070] Step S1011': Alternatively, receive a user instruction, read the configured field list data in the system, and obtain the field definition information.

[0071] Understandably, when configuring an EBCDIC file reading plugin developed based on the Kettle plugin's extension mechanism, a field list can be pre-configured according to user instructions. By reading the data in this field list, the field definition information can also be read.

[0072] Step S102: Read the file to be converted based on the preset byte length, and obtain the numerical information corresponding to each field in the file to be converted according to the field definition information to determine the field-level data.

[0073] It should be noted that this preset byte length can be a user-defined read byte length or an automatically generated read byte length based on the actual byte size of the file to be read. This preset byte length can also be configured simultaneously when configuring the EBCDIC file reading plugin.

[0074] In practice, the processing device can read the file to be converted according to a preset byte length, read the file content of a specified length at a time, and then, according to the field definition information, split the bytes of each line of the data in the file to be converted into the correct byte content and numerical information contained in each field based on the field width, field type, field format, and can also combine field grouping and field reference characteristics to determine the field-level data.

[0075] Step S20: Decode the field-level data using an encoded character set to obtain the decoded data.

[0076] It should be noted that an encoded character set can be a collection of code tables that contain the mapping relationship between EBCDIC encoded code point numbers and Unicode general characters. This encoded character set can include different encoded character set IDs (CCSIDs), such as code tables with CCSIDs of 935, 937, 37 or 1388.

[0077] It is understandable that when receiving the field-level data read based on the preset byte length, each field can be decoded according to its CCSID encoding format, and the correct byte content and numerical information corresponding to each field read can be parsed into the corresponding encoded actual content.

[0078] Step S30: Obtain the standardization processing function, and perform data cleaning on the decoded data according to the standardization processing function to obtain the target data.

[0079] It should be noted that this standardized processing function can be a function that users pre-set to filter and update decoded data based on the data processing needs of the current scenario or the historical characteristics of the original data in the current scenario.

[0080] In practice, the processing device calls a user-preset standardized processing function. This function cleans the decoded data containing the actual content, removing invalid and duplicate data and standardizing the data to obtain the target data.

[0081] Step S40: Generate a universal storage file based on the target data.

[0082] Understandably, this universal storage file can be a file using common Unicode encodings, such as GBK, UTF-8, and other mainstream encoding formats as its data storage format. This universal storage file can contain all valid data from the original EBCDIC format file to be converted. This universal storage file can be stored on the current processing device, allowing downstream applications to directly access the universal storage file when they need to retrieve the data stored in EBCDIC format from the file to be converted.

[0083] This embodiment reads the file to be converted based on the field definition table to obtain field-level data; decodes the field-level data using an encoded character set to obtain decoded data; obtains a standardized processing function and cleans the decoded data using the standardized processing function to obtain target data; and generates a general storage file based on the target data. Because this embodiment combines field definition information and reads the file to be converted based on a preset byte length to obtain field-level data (where the field definition data can be pre-configured from COBOL file parsing or EBCDIC reading plugins), and decodes the field-level data of the file to be converted using an encoded character set to obtain decoded data, and then cleans the decoded data using a standardized processing function, a general storage file is generated based on the obtained target data for downstream applications. This achieves data reading and conversion of the file and solves the problem of garbled text in the file.

[0084] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the encoded data processing method of the present invention.

[0085] Based on the above embodiments, in order to improve the data decoding rate after obtaining the encoded character set, step S20 includes:

[0086] Step S201: Split the field-level data into columns and generate decoding objects corresponding to each column of field-level data according to the encoded character set.

[0087] It should be noted that when receiving field-level data, in order to facilitate the decoding of the field-level data, that is, to decode the correct byte content and numerical information contained in each field, since CCSID is assigned to each column, the field-level data can be split into columns first, so that all the content in the above field-level data can be assigned to a single CCSID.

[0088] Understandably, after obtaining the field-level data split by column, one can obtain the CCSID of each column and construct the corresponding decoding object for each column based on the CCSID.

[0089] In practical implementation, field-level data can be split into columns based on the CCSID in the encoded character set. Then, decoding objects for each column can be constructed based on the CCSID, so that the actual value of the entire field-level data can be parsed based on the decoding objects.

[0090] Step S202: Obtain the numerical values ​​of each column field-level data and convert the numerical values ​​of each column field-level data into binary values.

[0091] In the actual implementation, the values ​​of each column can be obtained in the row order of the field-level data, and the values ​​of each column of field-level data can be converted into values ​​in binary representation so that the values ​​can be converted into actual values ​​later.

[0092] Step S203: Parse the binary value using the decoding object to obtain the decoded data.

[0093] In the specific implementation, the decoding object of the corresponding column constructed based on each CCSID can be obtained, and combined with the field-level values ​​of each column in binary numerical form, the binary content of the corresponding column can be parsed into the actual value through the decoding object to obtain the decoded data.

[0094] Furthermore, considering that the file to be converted contains a certain amount of data, and the processing device can only read a preset byte length at a time when reading the file to be converted, after decoding a certain line of field-level data, the field definition information of that line of field-level data and the parsed decoded data can be encapsulated into a mapping. Then, the reading and parsing of the next line of field-level data in the file to be read is repeated until the entire file to be read is read. Therefore, after step S203, the following is also included:

[0095] Step S204: Obtain the field definition information and combine it with the decoded data to generate a mapping result set.

[0096] It should be noted that after reading the entire contents of the file to be read and completing the conversion of the EBCDIC encoded data, the mapping result set can be obtained by integrating the field definition information of each line in the file with the corresponding decoded data encapsulation.

[0097] Accordingly, step S30 includes:

[0098] Step S30': Obtain the standardization processing function, and perform data cleaning on the decoded data in the mapping result set according to the standardization processing function to obtain the target data.

[0099] Furthermore, the mapping result set can be stored in the memory of the current processing device. The mapping result set can retain all the data in the original EBCDIC format file to be converted. Since the mapping result set retains field definition information and decoded data, if there is a data acquisition error when the subsequently generated general storage file is used by downstream applications, the data content in the generated general storage file can be checked and updated by obtaining the mapping result set. This improves the reliability and robustness of the file data obtained when the file to be converted is specifically decoded using the encoding data processing method in this embodiment in the current scenario.

[0100] This embodiment splits the field-level data into columns and generates decoding objects corresponding to each column of field-level data according to the encoded character set; it obtains the numerical values ​​of each column of field-level data and converts them into binary values; it then parses the binary values ​​using the decoding objects to obtain the decoded data. Furthermore, upon reading the entire file content and completing the conversion of EBCDIC encoded data, it integrates the mappings of field definition information and the corresponding decoded data in each row of the file to obtain a mapping result set. This enables the reading and parsing of EBCDIC encoded file data. Using CCSID to construct the decoding objects accurately identifies the actual values ​​of each field in the file to be converted, thus improving the decoding speed of EBCDIC encoded file data.

[0101] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the encoded data processing method of the present invention.

[0102] Based on the above embodiments, considering that there is invalid redundant data in all the data of the original EBCDIC format file to be converted, after obtaining the decoded data, the decoded data can be further processed to improve the effectiveness of the target data. Therefore, step S30' includes:

[0103] Step S301: Perform field-level filtering on each row of data in the decoded data according to preset row filtering conditions to obtain the filtered decoded data.

[0104] It should be noted that the preset row filtering conditions can be set by the user in advance based on the historical data characteristics of the file to be converted in the current scenario or the data processing requirements of the current scenario. Fields in each row can also be filtered according to the configuration of the field list. For example, if the source field of a row in the decoded data is [COL_A,COL_B,COL_C], and the field needs to be reconfigured to [COL_A,COL_C], then the source field can be filtered out using the row filtering conditions.

[0105] Understandably, filtering can also be performed based on the specific numerical values ​​in the data. For example, the field-level numerical value 'a' corresponding to field A can be compared with a preset numerical value 'x'. If 'a' > 'x', field A is retained; otherwise, field A is discarded.

[0106] Step S302: Perform format processing on the filtered and decoded data.

[0107] Understandably, this formatting process can include field renaming, field type conversion, and field order reordering. Field renaming can normalize the names of fields in the decoded data at the field level; field type conversion can, for example, convert string data and date data in the decoded data; field order reordering can, for example, if a row in the decoded data has source fields [COL_A,COL_B,COL_C] and needs to be reconfigured to [COL_A,COL_C], and the operation of filtering out the source fields by row filtering conditions fails, the source fields in that row can be rearranged by field order reordering to adjust the order of the fields in the row and obtain the ordered fields that meet the conditions.

[0108] Step S303: Clean the decoded data after format processing using a standardization processing function to obtain the target data.

[0109] Understandably, this standardized processing function can be a custom function set in Kettle using the QLexpress expression engine combined with user commands. This standardized function can perform data cleaning processes including replacing invisible characters in field content and removing trailing spaces. For example, when the obtained decoded data uses the common GBK encoding format, the custom function StrClnsGBK() can convert 33 invisible ASCII characters (\u00~\u1F and \u7F) to spaces under GBK encoding.

[0110] In practical implementation, when the processing device obtains the decoded data that has undergone the above line filtering and format processing, it can further call the pre-set standardized processing function to perform field-level data cleaning on the decoded data, obtain target data containing all valid data in the original file to be converted, improve the data reliability of the subsequently obtained general storage file, and reduce the resource consumption of redundant data.

[0111] Furthermore, considering the diversity of downstream applications, when the file to be converted is converted into target data, multiple different types of general storage files can be generated based on the target data to broaden the application scenarios of the data. The general storage files include: data files, control files, and metadata files.

[0112] It should be noted that the data file can store the actual data content of the target data. This data file can be stored according to a unified format specification, such as using common encoding formats like GBK or UTF. Specifically, 0x7C and 0x1C can be used as field separators during storage, and 0x0A as record separators.

[0113] The control file can store basic information about the file to be converted corresponding to the target data. For the file to be converted, i.e. the data file, read in a task, the control file can contain only 1 record and 6 fields: data file name (excluding directory), data file size (in bytes), number of records in the data file, data file character set, data file MD5 value (default is empty), and data file format encoding.

[0114] Metadata files can store field information of the files to be converted corresponding to the target data. These metadata files can be in XML format, and the field information can include: the version number of the file to be converted, the character set of the file to be converted, the source system encoding of the file to be converted, the alias of the file to be converted, the content description of the file to be converted, the field name, the field content description, the field type, the field length, the field precision, the target database type of the file to be converted, and the corresponding field type, field length, field precision, and whether the field can be null, etc.

[0115] Accordingly, after step S40, the following steps are included:

[0116] Step S41: Obtain the target storage database of the file to be converted.

[0117] Step S42: Filter the general storage files according to the target storage database to obtain the current storage file corresponding to the target storage database.

[0118] It should be noted that, due to the different types of downstream applications, the destinations of the general storage files generated from the target data converted from the EBCDIC encoded format data of the files to be converted vary. Therefore, the generated general storage files can be filtered and matched based on the different target storage databases that downstream applications need to retrieve the files to be converted, in order to further improve the data retrieval rate of the EBDIC format files to be converted.

[0119] refer to Figure 5 , Figure 5 This is a matching diagram between the general storage files and the target storage database in this embodiment. Figure 5 If downstream target storage database I only needs to obtain the actual data content of the file to be converted, the data file can be the current storage file corresponding to target storage database I; if downstream target storage database II only needs to obtain the basic information of the file to be converted, the control file can be the current storage file corresponding to target storage database II; if downstream target storage database III only needs to obtain the field information of the file to be converted, the metadata file can be the current storage file corresponding to target storage database III.

[0120] This embodiment performs field-level filtering on each row of data in the decoded data according to preset row filtering conditions to obtain filtered decoded data; the filtered decoded data is then formatted, including field renaming, field type conversion, and field order reordering; the formatted decoded data is cleaned using a standardized processing function to obtain target data, which improves the data reliability of the subsequently obtained general storage file and reduces the resource consumption of redundant data; and the general storage file generated from the target data, including data files, control files, and metadata files, is filtered based on the downstream target storage database, which helps to further improve the data retrieval rate of the EBDIC format files to be converted.

[0121] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing an encoded data processing program, which, when executed by a processor, implements the steps of the encoded data processing method described above.

[0122] refer to Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the encoded data processing device of the present invention.

[0123] like Figure 6 As shown, the encoded data processing apparatus proposed in this embodiment of the invention includes:

[0124] The file reading module 601 is used to read the file to be converted according to the field definition table and obtain field-level data.

[0125] Data decoding module 602 is used to decode the field-level data using an encoded character set to obtain decoded data;

[0126] The conversion and cleaning module 603 is used to obtain a standardized processing function and perform data cleaning on the decoded data according to the standardized processing function to obtain target data;

[0127] The data output module 604 is used to generate a general storage file based on the target data.

[0128] Furthermore, the file reading module 601 is also used to obtain field definition information, which includes field width, field type and field format; read the file to be converted based on a preset byte length, and obtain the numerical information corresponding to each field in the file to be converted according to the field definition information to determine the field-level data.

[0129] Furthermore, the file reading module 601 is also used to import COBOL files into the system, parse the COBOL files to obtain field definition information; or receive user instructions, read the configured field list data in the system, and obtain field definition information.

[0130] This embodiment reads the file to be converted based on the field definition table to obtain field-level data; decodes the field-level data using an encoded character set to obtain decoded data; obtains a standardized processing function and cleans the decoded data using the standardized processing function to obtain target data; and generates a general storage file based on the target data. Because this embodiment combines field definition information and reads the file to be converted based on a preset byte length to obtain field-level data (where the field definition data can be pre-configured from COBOL file parsing or EBCDIC reading plugins), and decodes the field-level data of the file to be converted using an encoded character set to obtain decoded data, and then cleans the decoded data using a standardized processing function, a general storage file is generated based on the obtained target data for downstream applications. This achieves data reading and conversion of the file and solves the problem of garbled text in the file.

[0131] Based on the first embodiment of the encoded data processing device of the present invention described above, a second embodiment of the encoded data processing device of the present invention is proposed.

[0132] In this embodiment, the data decoding module 602 is used to split the field-level data into columns and generate decoding objects corresponding to each column of field-level data according to the encoded character set; obtain the values ​​of each column of field-level data and convert the values ​​of each column of field-level data into binary values; and parse the binary values ​​through the decoding objects to obtain the decoded data.

[0133] Furthermore, the data decoding module 602 is also used to obtain the field definition information and, in combination with the decoded data, generate a mapping result set.

[0134] The conversion and cleaning module 603 is used to obtain a standardized processing function and perform data cleaning on the decoded data in the mapping result set according to the standardized processing function to obtain the target data.

[0135] Furthermore, the conversion and cleaning module 603 is also used to perform field-level filtering on each row of data in the decoded data according to preset row filtering conditions to obtain filtered decoded data; to perform format processing on the filtered decoded data, the format processing including field renaming, field type conversion, and field order arrangement; and to perform data cleaning on the format-processed decoded data through a standardized processing function to obtain target data.

[0136] The data output module 604 is used to obtain the target storage database of the file to be converted; and to filter the general storage files according to the target storage database to obtain the current storage file corresponding to the target storage database, wherein the general storage files include: data files, control files and metadata files.

[0137] Other embodiments or specific implementations of the coded data processing device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0139] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0141] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for processing encoded data, characterized in that, The method is applied to the Kettle system, and the method includes: Read the file to be converted from the field definition table to obtain field-level data; The field-level data is decoded using an encoded character set to obtain decoded data, including: The field-level data is split into columns, and a decoding object corresponding to each column of field-level data is generated according to the encoded character set; Obtain the values ​​of each column of field-level data according to the row order of the field-level data, and convert the values ​​of each column of field-level data into binary values; The decoded data is obtained by parsing the binary value using the decoding object. Obtain the field definition information and combine it with the decoded data to generate a mapping result set; Obtain a standardized processing function, and perform data cleaning on the decoded data in the mapping result set according to the standardized processing function to obtain the target data. The standardized processing function is a custom function pre-set by the expression engine QLexpress in the Kettle system in combination with user instructions, which is used for the data cleaning process of replacing invisible characters in the field content and deleting trailing spaces. Based on the target data, different types of general-purpose storage files are generated, including: data files, control files, and metadata files; The step of reading the file to be converted from the field definition table to obtain field-level data includes: Obtain field definition information, which includes field width, field type, and field format. The field definition information is obtained by parsing the field definition table in the COBOL file imported into the Kettle system, or by reading the configured field list data in the Kettle system. The file to be converted is read based on a preset byte length, and the numerical information corresponding to each field in the file to be converted is obtained according to the field definition information to determine the field-level data. The preset byte length is automatically generated based on the actual byte size of the file to be converted.

2. The encoded data processing method as described in claim 1, characterized in that, The process of obtaining field definition information includes: Import the COBOL file into the system and parse the COBOL file to obtain field definition information; Alternatively, it can receive user commands, read the configured field list data in the system, and obtain field definition information.

3. The encoded data processing method as described in claim 1, characterized in that, The step of obtaining a standardized processing function and cleaning the decoded data in the mapping result set according to the standardized processing function to obtain the target data includes: The decoded data is filtered at the field level according to the preset row filtering conditions to obtain the filtered decoded data. The filtered and decoded data is then processed in a format, including field renaming, field type conversion, and field order reordering. The decoded data, after being formatted, is cleaned using a standardized processing function to obtain the target data.

4. The encoded data processing method according to any one of claims 1 to 3, characterized in that, After generating different types of general-purpose storage files based on the target data, the process includes: Obtain the target storage database for the file to be converted; The target storage database is filtered in the general storage file to obtain the current storage file corresponding to the target storage database.

5. A data encoding processing apparatus, characterized in that, The encoded data processing device includes: The file reading module is used to read the file to be converted based on the field definition table and obtain field-level data. The data decoding module is used to decode the field-level data using an encoded character set to obtain the decoded data; The data decoding module is further configured to perform column splitting on the field-level data and generate decoding objects corresponding to each column of field-level data according to the encoded character set; obtain the values ​​of each column of field-level data according to the row order of the field-level data and convert the values ​​of each column of field-level data into binary values; parse the binary values ​​through the decoding objects to obtain decoded data; obtain the field definition information and generate a mapping result set in combination with the decoded data; The transformation and cleaning module is used to obtain a standardized processing function and clean the decoded data in the mapping result set according to the standardized processing function to obtain the target data. The standardized processing function is a custom function pre-set by the expression engine QLexpress in the Kettle system in combination with user commands. It is used for the data cleaning process of replacing invisible characters in the field content and deleting trailing spaces. The data output module is used to generate different types of general storage files based on the target data. The general storage files include: data files, control files, and metadata files. The file reading module is also used to obtain field definition information, which includes field width, field type, and field format. The field definition information is obtained by parsing the field definition table in the COBOL file imported into the Kettle system, or by reading the configured field list data in the Kettle system. The module reads the file to be converted based on a preset byte length and obtains the numerical information corresponding to each field in the file to be converted according to the field definition information to determine the field-level data.

6. A data encoding processing device, characterized in that, The device includes: a memory, a processor, and an encoded data processing program stored in the memory and executable on the processor, the encoded data processing program being configured to implement the steps of the encoded data processing method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores an encoded data processing program, which, when executed by a processor, implements the steps of the encoded data processing method as described in any one of claims 1 to 4.

Images