Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium, which can be used in the field of big data. The method comprises the following steps: obtaining user demand, calling corresponding source data from a source system database based on the user demand; analyzing the data quality of the source data when the source data is transmitted to a target system by using a file transmission protocol, obtaining a data exception type and a problem code, and judging whether to perform data repair based on the data exception type; the data exception type is a type defined in advance for expressing data errors; when it is determined to perform data repair, the problem code in the source data is converted by using a pre-defined code conversion mapping matrix table to obtain a target code, and the target code is transmitted to the target system; the pre-defined code conversion mapping matrix table is a mapping table customized based on the data exception type for code conversion. The method of the application performs data repair by code conversion in the data transmission process, thereby improving the data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Cross-platform systems are very complex systems, typically consisting of a host computer and multiple subsystems. The operating systems running on the host computer and the subsystems are often different. Therefore, during data exchange, due to differences in character set encoding between operating systems, encoding conversion is performed during data transmission or after receiving files. However, during encoding conversion, if the data source of the source system includes special function symbols, garbled characters may easily occur when the data is downloaded to the target system for File Transfer Protocol (FTP) encoding conversion.

[0003] In existing technologies, to solve the problem of garbled characters, each line of data, i.e. the character encoding of the entire record, is usually scanned when the source system generates the result data file. Abnormal encodings are found and repaired. Furthermore, the repaired result data file in the source system is transferred to the target system via FTP using a default encoding table.

[0004] However, the above method requires scanning the character encoding of each record before using FTP to transfer data, resulting in low data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium to solve the problem that existing methods require additional scanning of each character code of each record before transferring data using FTP, resulting in low data processing efficiency.

[0006] Firstly, this application provides a data processing method, including:

[0007] Obtain user requirements, and retrieve corresponding source data from the source system database based on the user requirements;

[0008] When the source data is transmitted to the target system using a file transfer protocol, the data quality of the source data is analyzed to obtain data exception types and problem codes, and a determination is made on whether to perform data repair based on the data exception types; the data exception types are predefined types used to describe data errors.

[0009] When data repair is determined, a predefined transcoding mapping matrix table is used to convert the problematic codes in the source data to obtain the target codes, and the target codes are transmitted to the target system; the predefined transcoding mapping matrix table is a mapping table customized based on the data exception type for encoding conversion.

[0010] Secondly, this application provides a data processing apparatus, comprising:

[0011] The acquisition module is used to acquire user requirements and retrieve corresponding source data from the source system database based on the user requirements.

[0012] The analysis module is used to analyze the data quality of the source data when the source data is transmitted to the target system using a file transfer protocol, obtain the data exception type and problem code, and determine whether to perform data repair based on the data exception type; the data exception type is a predefined type used to describe data errors;

[0013] The conversion module is used to convert the problematic codes in the source data into target codes using a predefined transcoding mapping matrix table when data repair is determined, and then transmit the target codes to the target system. The predefined transcoding mapping matrix table is a mapping table customized based on the data exception type for encoding conversion.

[0014] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0017] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0018] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any one of the first aspects.

[0019] The data processing method, apparatus, electronic device, and storage medium provided in this application obtain the required source data from the source system, analyze the data quality of the source data, obtain the data exception type of the erroneous data, and decide whether data repair can be performed by encoding mapping. If so, the erroneous encoding in the source data is converted into the correct target encoding and transmitted using a customized transcoding mapping matrix table. In this way, transcoding data repair is performed simultaneously during data transmission, improving data processing efficiency. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 This is a schematic diagram of the structure corresponding to an existing data processing method.

[0022] Figure 2 This is a schematic diagram of the structure corresponding to a data processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0024] Figure 4 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0026] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of this application.

[0027] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] It should be noted that the data processing method, apparatus, electronic device and storage medium provided in this application can be used in the field of big data, or in any field other than big data. This application does not limit the application field of the data processing method, apparatus, electronic device and storage medium.

[0031] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0033] The following is an explanation of the technical terms used in this application.

[0034] Cross-platform: This can refer to the information exchange between different hardware environments and operating systems.

[0035] Cross-platform system: includes source system and target system. The source system can be the host core system, and the target system can be the peripheral system, i.e., non-core system. Non-core system can include operating systems such as WINDOWS and LINUX; the host core system is a commercial operating system (ZOS).

[0036] EBCDIC (Extended Binary Coded Decimal Interchange Code) encoding: This refers to an extended binary code decimal interchange code. The English letters in EBCDIC encoding are not arranged continuously; there are multiple breaks in the middle.

[0037] ASCII (American Standard Code for Information Interchange) encoding refers to a method that uses a specified combination of 7 or 8 binary numbers to represent 128 or 256 possible characters. It can use 7 binary numbers (with the remaining 1 binary bit set to 0) to represent all uppercase and lowercase letters, numbers 0 to 9, punctuation marks, and special control characters.

[0038] Cross-platform data transfer is typically in text format, transmitted as lines of fixed format and length. During cross-platform data exchange, encoding differences between operating systems necessitate encoding conversion during data transmission or after file reception. Sometimes, due to data quality issues in the source system or inconsistent input standards, additional processing of special characters is required after data transcoding, significantly reducing efficiency.

[0039] For example, if the source system's data source is negligent in input control, some online data from the front end or external systems may contain symbols such as carriage returns and line feeds. These function symbols will not be wrapped into two lines in the host's file storage, but when downloaded to the target system for FTP transcoding, they will turn one line of data in the host file into two lines of data, making it impossible for the target system to load the data according to the agreed data format. Therefore, it is necessary to process these function symbols, such as by using a special program in the source or target system to process them.

[0040] In one possible implementation, Figure 1 This is a schematic diagram of the structure corresponding to an existing data processing method, such as... Figure 1 As shown, when the source system generates the result data file, it typically scans each line of data, i.e., the character encoding of the entire record, finds abnormal encodings and repairs them. Further, it uses a default encoding table via FTP to transfer the repaired result data file from the source system to the target system. For example, if an abnormal encoding of '0D' (the encoding value of a carriage return on the host) is found, it might be converted to a space '40' (the encoding value of a space on the host) during the subsequent conversion. Therefore, '0D' is repaired to prevent line breaks during subsequent FTP operations.

[0041] The source system uses EBCDIC encoding, while the target system uses ASCII encoding.

[0042] However, the above method requires scanning the character encoding of each record before using FTP to transfer data, resulting in low data processing efficiency.

[0043] To address the aforementioned issues, this application provides a data processing method that can specially handle non-standard characters in certain scenarios, reducing workload without affecting data processing efficiency. Specifically, for some common data errors, a specially customized encoding table can be used during FTP transfer to perform data repair, and then the repaired data can be transferred to the target system; for example, Figure 2 This is a schematic diagram of the structure corresponding to a data processing method provided in an embodiment of this application, such as... Figure 2 As shown, the required source data, i.e., the corresponding EBCDIC encoding, is obtained from the source system. Further, the data quality of the EBCDIC encoding is analyzed to find erroneous data. Then, a specially customized transcoding table is used to repair the data, converting the erroneous encoding in the EBCDIC encoding into the correct target encoding, i.e., the corresponding ASCII encoding. Further, the data (ASCII encoding) is transferred to the target system via FTP. In this way, transcoding and data repair are performed simultaneously during data transmission, improving data processing efficiency.

[0044] It should be noted that after the target system receives the ASCII encoding, it can load the ASCII encoding based on a predefined data format and perform format conversion to obtain the required target data. The predefined data format is a pre-agreed format that can be used to load ASCII encoding. This application embodiment does not limit the specific format corresponding to the predefined data format.

[0045] For example, Figure 3 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 3 As shown, the application scenario includes: user terminal device 301, host core system 302 and peripheral system 303. In the host core system 302, business data can be transmitted to different peripheral systems 303.

[0046] Specifically, the host core system 302 can obtain user requirements based on the user's terminal device 301. Furthermore, the host core system 302 can obtain the corresponding data file from the database based on the user requirements. Furthermore, during the process of the host core system 302 transmitting the data file to the peripheral system 303, it can use a customized transcoding mapping matrix table to convert the incorrect encoding in the data file into the correct target encoding and transmit the correct target encoding to the peripheral system 303. Correspondingly, after receiving the target encoding, the peripheral system 303 can load the target encoding based on the agreed data format and perform format conversion to obtain the required target data file.

[0047] It should be noted that the host core system 302, as the main execution entity, can correspond to a device that can be displayed and operated. Accordingly, the host core system 302 can find the corresponding business function based on the user's touch operation on the device, and then generate a data file based on the business function. This application embodiment does not specifically limit the method of obtaining the data file.

[0048] Optionally, the aforementioned terminal equipment can be either a wireless terminal or a wired terminal. A wireless terminal can be a device that provides voice and / or other service data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a Radio Access Network (RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) or a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device, which exchanges voice and / or data with the RAN. For example, a wireless terminal can also be a Personal Communication Service (PCS) phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), or other similar devices. A wireless terminal can also be referred to as a system, subscriber unit, subscriber station, mobile station, mobile, remote station, remote terminal, access terminal, user terminal, user agent, user device, or user equipment; no specific terminology is used here. Optionally, the aforementioned terminal devices can be smartphones, tablets, or other similar devices.

[0049] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0050] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, the data processing method includes the following steps:

[0051] S401. Obtain user requirements and retrieve corresponding source data from the source system database based on the user requirements.

[0052] In this embodiment of the application, user requirements may correspond to different business functions, that is, the functional requirements corresponding to the user's need to handle different businesses or the system's corresponding business processing; the user requirements may be determined based on the actual situation of the application scenario or data, and may be input by the user or called by the system from the database based on the application scenario. This embodiment of the application does not make specific limitations on this.

[0053] In this step, after obtaining user requirements, the source system can generate data files based on the data generation module in the source system according to business functions, that is, call the corresponding source data from the source system database.

[0054] S402. When transmitting the source data to the target system using a file transfer protocol, the data quality of the source data is analyzed to obtain the data exception type and problem code, and a determination is made based on the data exception type whether to perform data repair; the data exception type is a predefined type used to describe data errors.

[0055] In this embodiment of the application, the data exception type can refer to the data type corresponding to the abnormal encoded value. The abnormal encoded value corresponds to the problem code, which may include non-standard characters, special characters, functional characters, etc. This embodiment of the application does not limit the specific form of the problem code, such as a carriage return, line feed, or "$" character, etc.

[0056] In this step, when the source data is transmitted to the target system using the file transfer protocol, the data generation module can also analyze the data quality of the source data, that is, determine whether the EBCDIC encoding value corresponding to the source data has at least one identical value with the encoding value in the predefined data table containing the problem encoding. If so, based on the data exception type corresponding to at least one identical value in the data table, the erroneous data and the erroneous data type can be obtained, that is, the data exception type and problem encoding corresponding to the source data, and based on the data exception type, it can be determined whether the data can be repaired by encoding mapping.

[0057] S403. When it is determined that data repair is to be performed, the problematic code in the source data is converted using a predefined transcoding mapping matrix table to obtain the target code, and the target code is transmitted to the target system; the predefined transcoding mapping matrix table is a mapping table customized based on the data exception type for encoding conversion.

[0058] In this embodiment, a predefined transcoding mapping matrix table is used to convert E code (EBCDIC encoding) to A code (ASCII encoding) during data transmission. The predefined transcoding mapping matrix table is generated based on the actual situation of the data and customized conversion rules. Different transcoding mapping matrix tables can be defined for data in different application scenarios, or a unified transcoding mapping matrix table can be defined. This embodiment does not make specific limitations on this. As shown in Table 1, it is a customized transcoding mapping matrix table.

[0059] Table 1

[0060]

[0061] In Table 1, the second row and last column correspond to the conversion codes, which are the basis for converting E code to A code. For example, taking the E code value "12" in the second row as an example, according to the principle of coordinate point intersection, the first digit "2" of the value "02" in the second row and the second digit "1" of the value "10" in the last column are taken to form the conversion value "21", which is the corresponding A code value. Similarly, taking the E code value "6C" in the fourth row as an example, according to the principle of coordinate point intersection, the first digit "5" of the value "05" in the second row and the second digit "2" of the value "20" in the last column are taken to form the conversion value "52", which is the corresponding A code value. Other E code to A code conversions are similar to the above methods and will not be described in detail here. Each E code value corresponds to a character value.

[0062] In this step, once it is determined that data repair can be performed using encoding mapping, the transmission module in the source system uses the customized transcoding mapping matrix table (as shown in Table 1) to convert some erroneous encodings in the source data into correct target encodings, and then transmits the target encodings to the target system.

[0063] It is understood that during FTP data transfer, the source data without errors in its encoding can also be converted and transmitted to the target system. This can be done using a predefined transcoding mapping matrix table, which contains normal E-code to A-code rules; alternatively, an existing custom transcoding table can be used. This application does not specifically limit the specific method used in this embodiment.

[0064] Therefore, the embodiments of this application can obtain the required source data from the source system, analyze the data quality of the source data, obtain the data exception type of the erroneous data, and decide whether data repair can be performed by encoding mapping. If so, the erroneous encoding in the source data is converted into the correct target encoding and transmitted using the customized transcoding mapping matrix table. In this way, transcoding data repair is performed simultaneously during data transmission, which improves data processing efficiency.

[0065] Optionally, the customization process of the predefined transcoding mapping matrix table includes:

[0066] The application scenario for obtaining source data is determined, and transformation rules are customized based on the application scenario. The transformation rules include data exception types, problem code values, and transformation code values ​​corresponding to the problem code values.

[0067] A transcoding mapping matrix table is generated based on the conversion rules, and the transcoding mapping matrix table is verified to obtain a predefined transcoding mapping matrix table.

[0068] In this embodiment of the application, the conversion rule is used to record the rules for converting E code to A code. Based on this conversion rule, an FTP transcoding table can be customized for data exception types corresponding to different application scenarios. The conversion rule includes data exception type, problem code value, and the conversion code value corresponding to the problem code value, which respectively correspond to character value, problematic E code value, and the converted correct A code value, as shown in Table 2:

[0069] Table 2

[0070]

[0071]

[0072] The first column corresponds to the data exception type, which describes the characters. The data exception types include low value type, special symbol type, high value type, and paragraph symbol type. The low value type is a character type where all values ​​are zero. The special symbol type includes: serial number type, mathematical symbol type, punctuation mark type, unit symbol type, and arrow symbol type, such as "$, \, ~" shown in Table 2, and "+, -, ×, ÷, ±, %" not shown in Table 2. The high value type is a character type where all values ​​are 1, corresponding to 11111111 in binary and FF in hexadecimal, where F represents the largest single digit in hexadecimal. The paragraph symbol type includes carriage return type (CR), line feed type (LF), etc. Wherein, the character corresponding to CR is "\r", and the character corresponding to LF is "\n".

[0073] The second column is the EBCDIC encoding value of the host, the third column is the customized E code to ASCII encoding value, and the fourth column is the corresponding E code to A code encoding value in the original transcoding table. Using the original transcoding table for encoding conversion can easily lead to encoding conversion confusion, such as converting '0D' to '0D', which reduces recognition efficiency.

[0074] It should be noted that the embodiments of this application do not specifically limit the types and number of symbol types corresponding to each data exception type. The above is only an example. By configuring and setting the data exception types and corresponding encoding values ​​in the transcoding mapping matrix table, the data processing logic does not need to be modified, which greatly improves the stability of data processing.

[0075] Therefore, this application can continuously enrich the data exception types to meet more application scenarios, thereby formulating corresponding transcoding mapping matrix tables and improving the flexibility of transcoding mapping matrix table applications.

[0076] In this step, after customizing the conversion rules based on the application scenario of the source data and generating the transcoding mapping matrix table using the conversion rules, the generated transcoding mapping matrix table can be verified to check whether the transcoding accuracy of the generated transcoding mapping matrix table meets the requirements, or whether the corresponding transcoding accuracy is higher than the preset threshold, so as to achieve a good conversion effect.

[0077] It should be noted that the method for verifying the transcoding mapping matrix table in this application embodiment is not specifically limited. It can be compared with manual transcoding to obtain the transcoding accuracy, or it can be compared with the method of conversion using the original transcoding table to obtain the transcoding accuracy, or it can be verified based on a specific verification algorithm to obtain the transcoding accuracy, so as to judge the quality of the conversion effect. This application embodiment does not specifically limit this.

[0078] Therefore, the embodiments of this application can use a customized transcoding mapping matrix table to fix data quality problems, improve the flexibility of the application, and evaluate the quality of the conversion effect by verifying the transcoding mapping matrix table.

[0079] Optionally, the transcoding mapping matrix table is validated to obtain a predefined transcoding mapping matrix table, including:

[0080] Multiple types of source data are acquired, and the multiple types of source data are copied to obtain verification data and comparison data. The verification data is used for encoding conversion based on the transcoding mapping matrix table to obtain a first code. The comparison data is used for manual encoding conversion to obtain a second code, so as to verify whether the first code converted by the transcoding mapping matrix table is correct.

[0081] The transcoding mapping matrix table is verified using the verification data and the comparison data to obtain the verification result, and a predefined transcoding mapping matrix table is generated based on the verification result.

[0082] In this embodiment of the application, the various types of source data can refer to different data types corresponding to different business functions, such as banking data types, teaching data types, power data types, communication data types, medical data types, etc. This embodiment of the application does not specifically limit the types of source data, and the source data can correspond to different business functions in various industries.

[0083] In this step, the source data is copied to obtain two or more identical datasets. These datasets can be encoded using different conversion methods, and the results are compared to verify the success rate of the conversion. For example, taking two datasets as verification data and comparison data, the first encoding is obtained based on the verification data and using a transcoding mapping matrix table. The second encoding is obtained by manually encoding based on the comparison data. The success rate of the first and second encoding conversions is compared to verify the effectiveness of the transcoding mapping matrix table in encoding conversion, thereby generating a predefined transcoding mapping matrix table.

[0084] Optionally, if it is determined through comparison that the transcoding mapping matrix table has incorrect encoding conversions, the transcoding mapping matrix table can be manually adjusted, that is, the transcoding mapping matrix table can be updated and configured to generate a transcoding mapping matrix table with better conversion effect, that is, a predefined transcoding mapping matrix table.

[0085] Therefore, embodiments of this application can verify the transcoding mapping matrix table to improve the stability of encoding conversion using the transcoding mapping matrix table.

[0086] Optionally, the data quality of the source data is analyzed to obtain data exception types and problem codes, including:

[0087] Obtain a data table that includes multiple data exception types; the data table includes pre-statistically collected problem codes of various types and the data exception types corresponding to the problem codes; each data exception type corresponds to a problem code;

[0088] Read the data from the source data to obtain the encoding value corresponding to the source data, and compare the encoding value corresponding to the source data with the problem encoding in the data table;

[0089] When it is determined that the source data contains a problem code in the data table, the corresponding data exception type and problem code are obtained.

[0090] In this embodiment of the application, the data quality of historical source data corresponding to different services is analyzed extensively, and the data exception types corresponding to the source data are summarized. Furthermore, the summarized data exception types and the corresponding problematic codes of the data exception types are aggregated to obtain a data table. The data table is used to compare with the code values ​​corresponding to the source data to verify whether there are problematic codes in the source data.

[0091] In this step, by checking whether the problem code in the data table corresponds to the code value in the source data, that is, if the code value in the source data corresponds to one or more problem codes in the data table, the data exception type and problem code corresponding to the erroneous data in the source data can be determined.

[0092] Therefore, the embodiments of this application can utilize data tables to analyze problems, find problem codes, and simplify the processing flow.

[0093] Optionally, determining whether to perform data repair based on the data exception type includes:

[0094] Determine whether the data exception type exists in the predefined transcoding mapping matrix table;

[0095] If so, then proceed with data repair;

[0096] If not, a prompt message is generated based on the data exception type to remind users to re-customize the predefined transcoding mapping matrix table.

[0097] In this embodiment, the prompt message can be used to indicate that the data exception type corresponding to the source data is not in the predefined transcoding mapping matrix table, that is, the predefined transcoding mapping matrix table lacks a configuration for a certain data exception type. Based on this prompt message, the user can be reminded to re-customize the defined transcoding mapping matrix table. The problem code and conversion rules corresponding to the missing data exception type can be added, thereby updating the defined transcoding mapping matrix table. Alternatively, all parameters and conversion rules in the defined transcoding mapping matrix table can be reconfigured. This embodiment does not specifically limit this.

[0098] For example, with Figure 3Taking the application scenario shown as an example, the host core system 302 can determine whether the data exception type corresponding to the data file exists in the predefined transcoding mapping matrix table; if it exists, the predefined transcoding mapping matrix table is used to repair the data; when it is determined that the data exception type corresponding to the data file does not exist in the predefined transcoding mapping matrix table, a prompt message can be generated based on the data exception type corresponding to the data file and sent to the user's terminal device 301 in the form of an SMS, reminding the user to re-customize the predefined transcoding mapping matrix table; the content of the prompt message can be "Data exception type 1 is missing, please configure".

[0099] It should be noted that the embodiments of this application do not specifically limit the form of sending the prompt message or the content of the display. The above is only an example. For example, it can also be sent in the form of a message box.

[0100] Therefore, the embodiments of this application can repair the number quality problem by using a customized transcoding mapping matrix table, and the transcoding mapping matrix table can be updated with parameters to improve the versatility of the application.

[0101] Optionally, transmitting the target code to the target system includes:

[0102] The target code is encrypted using an encryption algorithm, and the encrypted target code is transmitted to the target system via the file transfer protocol.

[0103] In this application embodiment, the encryption algorithm can refer to an algorithm that processes files or data that were originally plaintext. The encryption algorithm can make them into an unreadable piece of code called "ciphertext". This application embodiment does not specifically limit the content and form of the "ciphertext".

[0104] In this step, the target code can be encrypted by calling an encryption algorithm, such as the SM2 elliptic curve public key cryptography algorithm (SM is the pinyin abbreviation for commercial cryptography), thereby generating ciphertext, and then transmitting the ciphertext to the target system through a file transfer protocol; the encryption algorithm may include asymmetric encryption algorithms, symmetric encryption algorithms, hash algorithms, etc., and the specific algorithm corresponding to the encryption algorithm in this application embodiment is not limited.

[0105] It should be noted that each encryption algorithm has a corresponding inverse process, namely the decryption algorithm. The decryption algorithm is an algorithm that converts the encoded information into the original data. The specific algorithm corresponding to the decryption algorithm in this application embodiment is not limited, but the decryption algorithm used in this application needs to correspond to the encryption algorithm.

[0106] Furthermore, after the target system receives the encrypted target code, it can decrypt it based on the decryption algorithm to obtain the target code, load the target code according to the agreed data format, and perform format conversion to obtain the required target data file.

[0107] Therefore, the embodiments of this application can encrypt data transmission, thereby improving the security of data transmission.

[0108] In the foregoing embodiments, the data processing method provided by the embodiments of this application has been described. To implement the functions of the methods provided by the embodiments of this application, the electronic device serving as the execution subject may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0109] For example, Figure 5 This is a schematic diagram of a data processing device provided in an embodiment of this application. The device includes: an acquisition module 510, an analysis module 520, and a conversion module 530; wherein, the acquisition module 510 is used to acquire user requirements and retrieve corresponding source data from the source system database based on the user requirements;

[0110] The analysis module 520 is used to analyze the data quality of the source data when the source data is transmitted to the target system using a file transfer protocol, obtain the data exception type and problem code, and determine whether to perform data repair based on the data exception type; the data exception type is a predefined type used to describe data errors;

[0111] The conversion module 530 is used to convert the problematic code in the source data into a target code by using a predefined transcoding mapping matrix table when data repair is determined, and then transmit the target code to the target system; the predefined transcoding mapping matrix table is a mapping table customized based on the data exception type for encoding conversion.

[0112] Optionally, the device further includes a customization module, which includes a customization unit and a verification unit;

[0113] Specifically, the customization unit is used to obtain the application scenario of the source data and customize the transformation rules based on the application scenario; the transformation rules include data exception types, problem code values, and transformation code values ​​corresponding to the problem code values;

[0114] The verification unit is used to generate a transcoding mapping matrix table based on the conversion rules, and to verify the transcoding mapping matrix table to obtain a predefined transcoding mapping matrix table.

[0115] Optionally, the verification unit is specifically used for:

[0116] Multiple types of source data are acquired, and the multiple types of source data are copied to obtain verification data and comparison data. The verification data is used for encoding conversion based on the transcoding mapping matrix table to obtain a first code. The comparison data is used for manual encoding conversion to obtain a second code, so as to verify whether the first code converted by the transcoding mapping matrix table is correct.

[0117] The transcoding mapping matrix table is verified using the verification data and the comparison data to obtain the verification result, and a predefined transcoding mapping matrix table is generated based on the verification result.

[0118] Optionally, the analysis module 520 includes an analysis unit and a judgment unit; wherein, the analysis unit is used for:

[0119] Obtain a data table that includes multiple data exception types; the data table includes pre-statistically collected problem codes of various types and the data exception types corresponding to the problem codes; each data exception type corresponds to a problem code;

[0120] Read the data from the source data to obtain the encoding value corresponding to the source data, and compare the encoding value corresponding to the source data with the problem encoding in the data table;

[0121] When it is determined that the source data contains a problem code in the data table, the corresponding data exception type and problem code are obtained.

[0122] Optionally, the determination unit is used for:

[0123] Determine whether the data exception type exists in the predefined transcoding mapping matrix table;

[0124] If so, then proceed with data repair;

[0125] If not, a prompt message is generated based on the data exception type to remind users to re-customize the predefined transcoding mapping matrix table.

[0126] Optionally, the data exception types include: low value type, special symbol type, high value type, and paragraph symbol type; the low value type is a character type where all values ​​are zero; the high value type is a character type where all values ​​are 1.

[0127] Optionally, the conversion module 530 is specifically used for:

[0128] The target code is encrypted using an encryption algorithm, and the encrypted target code is transmitted to the target system via the file transfer protocol.

[0129] The specific implementation principle and effects of the data processing device provided in this application embodiment can be found in the relevant descriptions and effects of the above embodiments, and will not be elaborated further here.

[0130] This application also provides a schematic diagram of the structure of an electronic device. Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of this application.

[0131] The electronic device 600 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc., and can be used to achieve the above. Figures 2-4 The method described in the illustrated embodiment.

[0132] like Figure 6 As shown, the electronic device 600 may include at least one processor 610 for implementing Figures 2-4 The method described in the illustrated embodiment.

[0133] For example, the processor 610 can be used to acquire user requirements, retrieve corresponding source data from the source system database based on the user requirements, analyze the data quality of the source data when transmitting the source data to the target system using a file transfer protocol, obtain data exception types and problem codes, and determine whether to perform data repair based on the data exception types; the data exception types are predefined types used to describe data errors; when it is determined that data repair should be performed, the problem codes in the source data are encoded using a predefined transcoding mapping matrix table to obtain target codes, and the target codes are transmitted to the target system; the predefined transcoding mapping matrix table is a mapping table customized based on the data exception types for encoding conversion. See the detailed description in the method example for further details, which will not be repeated here.

[0134] The electronic device 600 may also include at least one memory 620 for storing program instructions and / or data. The memory 620 is configured to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. The memory 620 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] The memory 620 may be coupled to the processor 610. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 610 may operate in conjunction with the memory 620. The processor 610 may execute program instructions stored in the memory 620. At least one of the at least one memory may be included in the processor.

[0136] The electronic device 600 may further include a receiver 630 and a transmitter 640 for communicating with other devices via a transmission medium, thereby enabling the electronic device 600 to communicate with other devices. The processor 610 may use the receiver 630 to receive data and / or information, and the transmitter 640 to transmit data and / or information, and to implement... Figures 2-4 The method described in the illustrated embodiment.

[0137] The specific connection medium between the processor 610, memory 620, receiver 630 and transmitter 640 described above is not limited in the embodiments of this application.

[0138] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0139] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 620 including instructions, which can be executed by a processor 610 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0140] This application also provides a computer program product, the computer program product comprising: a computer program (also referred to as code or instructions), which, when run, causes a computer to perform the following actions: Figures 2-4 The method described in the illustrated embodiment.

[0141] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is executed, it causes the computer to perform actions such as... Figures 2-4 The method described in the illustrated embodiment.

[0142] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface; the indirect coupling or communication connection of apparatus or modules may be electrical, mechanical, or other forms.

[0143] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0145] In the above embodiments, the functions of each functional module can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Video Discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0146] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0148] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain user requirements, and retrieve corresponding source data from the source system database based on the user requirements; When transmitting the source data to the target system using a file transfer protocol, the data quality of the source data is analyzed to obtain data exception types and problem codes, and it is determined whether the data exception type exists in a predefined transcoding mapping matrix table. If it does, data repair is performed; if not, a prompt message is generated based on the data exception type to remind users to re-customize the predefined transcoding mapping matrix table. The data exception type is a predefined type used to describe data errors. The data exception types include: low value type, special symbol type, high value type, and paragraph symbol type. The low value type is a character type where all values ​​are zero; the high value type is a character type where all values ​​are 1. When data repair is determined, during the encoding conversion process of the file transfer protocol, a predefined transcoding mapping matrix table is used to convert the problematic encoding in the source data to obtain the target encoding, and the target encoding is transmitted to the target system; the predefined transcoding mapping matrix table is a mapping table customized based on the data exception type and used for encoding conversion during the encoding conversion process of the file transfer protocol; The customization process of the predefined transcoding mapping matrix table includes: obtaining the application scenario of the source data, and customizing the conversion rules based on the application scenario; the conversion rules include data exception types, problem coding values, and conversion coding values ​​corresponding to the problem coding values; generating a transcoding mapping matrix table based on the conversion rules, and verifying the transcoding mapping matrix table to obtain the predefined transcoding mapping matrix table.

2. The method according to claim 1, characterized in that, The transcoding mapping matrix table is verified to obtain a predefined transcoding mapping matrix table, including: Multiple types of source data are acquired, and the multiple types of source data are copied to obtain verification data and comparison data. The verification data is used for encoding conversion based on the transcoding mapping matrix table to obtain a first code. The comparison data is used for manual encoding conversion to obtain a second code, so as to verify whether the first code converted by the transcoding mapping matrix table is correct. The transcoding mapping matrix table is verified using the verification data and the comparison data to obtain the verification result, and a predefined transcoding mapping matrix table is generated based on the verification result.

3. The method according to claim 1, characterized in that, The data quality of the source data is analyzed to obtain data exception types and problem codes, including: Obtain a data table that includes multiple data exception types; the data table includes pre-statistically collected problem codes of various types and the data exception types corresponding to the problem codes; each data exception type corresponds to a problem code; Read the data from the source data to obtain the encoding value corresponding to the source data, and compare the encoding value corresponding to the source data with the problem encoding in the data table; When it is determined that the source data contains a problem code in the data table, the corresponding data exception type and problem code are obtained.

4. The method according to any one of claims 1-3, characterized in that, Transmitting the target code to the target system includes: The target code is encrypted using an encryption algorithm, and the encrypted target code is transmitted to the target system via the file transfer protocol.

5. A data processing apparatus, characterized in that, The apparatus is used to implement the data processing method according to any one of claims 1-4, and the apparatus comprises: The acquisition module is used to acquire user requirements and retrieve corresponding source data from the source system database based on the user requirements. The analysis module is used to analyze the data quality of the source data when the source data is transmitted to the target system using a file transfer protocol, obtain the data exception type and problem code, and determine whether to perform data repair based on the data exception type; the data exception type is a predefined type used to describe data errors; The conversion module is used to convert the problematic codes in the source data into target codes using a predefined transcoding mapping matrix table when data repair is determined, and then transmit the target codes to the target system. The predefined transcoding mapping matrix table is a mapping table customized based on the data exception type for encoding conversion.

6. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 4.

Images