Data processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a data processing method, device, electronic device and storage medium. The method includes: obtaining raw data of an enumerated data type; determining first identification data according to the raw data; the raw data is uniquely associated with the first identification data; if it is determined that the number of raw data and the number of bits of the first identification data meet the preset constraint condition, the raw data is converted into second identification data according to the preset conversion rule; the raw data is uniquely associated with the second identification data, and the number of raw data matches the number of bits of the second identification data. By executing the technical solution provided by the embodiment of the present invention, it is possible to save the computing and storage space of the computer and improve the computing efficiency and retrieval efficiency.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, device, electronic device, and storage medium. Background Art

[0002] Extract-Transform-Load (ETL) is the process of extracting, cleaning and transforming the data of the business system and loading it into the data warehouse. The purpose is to integrate the scattered, messy and non-standard data in the enterprise and provide an analytical basis for the enterprise's decision-making. The design of ETL is divided into three parts: data extraction, data cleaning and transformation, and data loading. It is a very important part of building a data warehouse and data mining.

[0003] Related Technology In the process of processing big data, the value of the processed data field is an enumeration type, which is called an enumeration field, also known as a dictionary field. The value range of a dictionary field is generally defined to include a dictionary code, a dictionary value, and corresponding description information, wherein the dictionary code is generally defined as a numeric type (decimal or hexadecimal), and the dictionary value is usually a text expression. In addition to being easy to process, the dictionary code is generally defined to be "human readable and understandable". For example, the national administrative division code defined in the national standard is expressed in 6 decimal digits, and the 6 digits are divided into three regions, namely: the first two digits represent the province / autonomous region / municipality, the middle two digits represent the city, and the last two digits represent the district / county. The problem faced by this setting is that it will cause the dictionary code to be relatively long. When the dictionary code is involved in the data processing process, it will cause the system to occupy too much computing and storage resources, which will lead to resource waste and low data processing efficiency. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device, electronic device and storage medium, which can save the computing and storage space of a computer and improve the computing efficiency and retrieval efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising: obtaining original data whose data type is an enumeration type;

[0006] Determining first identification data according to the original data; wherein the original data is uniquely associated with the first identification data;

[0007] If it is determined that the amount of original data and the number of bits of the first identification data meet the preset constraints, the original data is converted into second identification data according to a preset conversion rule; wherein the original data is uniquely associated with the second identification data, and the amount of original data matches the number of bits of the second identification data.

[0008] In a second aspect, an embodiment of the present invention further provides a data processing device, the device comprising: an original data acquisition module, used to acquire original data whose data type is an enumeration type;

[0009] A first identification data determination module, configured to determine first identification data according to the original data; wherein the original data is uniquely associated with the first identification data;

[0010] A data conversion module, used to convert the original data into second identification data according to a preset conversion rule if it is determined that the amount of original data and the number of bits of the first identification data meet preset constraints; wherein the original data is uniquely associated with the second identification data, and the amount of original data matches the number of bits of the second identification data.

[0011] In a third aspect, an embodiment of the present invention further provides an electronic device, the device comprising:

[0012] one or more processors;

[0013] a storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the embodiments of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method as described in any one of the embodiments of the present invention.

[0016] The technical solution provided by the embodiment of the present invention obtains raw data of an enumeration type; determines first identification data according to the raw data; wherein the raw data is uniquely associated with the first identification data; if it is determined that the number of raw data and the number of bits of the first identification data meet the preset constraint condition, the raw data is converted into second identification data according to a preset conversion rule; wherein the raw data is uniquely associated with the second identification data, and the number of raw data matches the number of bits of the second identification data. By executing the technical solution provided by the embodiment of the present invention, it is possible to save computing and storage space and improve computing efficiency and retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0018] Figure 2 is a flow chart of another data processing method provided by an embodiment of the present invention;

[0019] Figure 3a It is a schematic diagram of the structure of a data processing system in the prior art;

[0020] Figure 3b is a schematic diagram of the structure of a data processing system applicable to an embodiment of the present invention;

[0021] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention. The method can be executed by a data processing device, the device can be implemented by software and / or hardware, and the device can be configured in an electronic device for data processing. The method is applied to the scenario of data processing of enumeration fields of big data. Figure 1 As shown, the technical solution provided by the embodiment of the present invention specifically includes:

[0025] S110: Acquire original data whose data type is an enumeration type.

[0026] Among them, the ETL process is the process of extracting, cleaning and transforming the data of the source subsystem and loading it into the target subsystem. The source subsystem can be a business system, and the target subsystem can be a data warehouse. The original data comes from the source subsystem, that is, the input of the enumeration field in the ETL process. The original data can be the dictionary encoding of the enumeration field in the business system. For example, the original data can be a national administrative division, that is, a district / county in a city in a province / autonomous region / municipality.

[0027] S120: Determine first identification data according to the original data.

[0028] The original data is uniquely associated with the first identification data.

[0029] Specifically, the first identification data may be a dictionary encoding of the enumeration class field in the target subsystem, that is, the output of the enumeration class field during the ETL processing. The first identification data may be a reference to a national standard, such as the national administrative division code generally references a national standard. The first identification data may be a reference to an industry standard. The first identification data may also be a reference to an enterprise standard. The first identification data may be set according to actual needs. The data type of the first identification data may be a short type, the data type of the first identification data may be an Integer type, the data type of the first identification data may be a string type, and the data type of the first identification data may be set according to actual needs. If the reference standard of the first identification data is determined in this scheme, the first identification data may be determined based on the original data, and the original data corresponds one-to-one to the first identification data.

[0030] S130: If it is determined that the amount of the original data and the number of bits of the first identification data meet a preset constraint condition, converting the original data into second identification data according to a preset conversion rule.

[0031] The original data is uniquely associated with the second identification data, and the number of the original data matches the number of bits of the second identification data.

[0032] Exemplarily, taking the original data as a national administrative division as an example, the number of national administrative divisions can be determined. Taking the first identification data citing the national standard as an example, the first identification data is the first six digits of the personal ID number. The preset constraint condition can be that the maximum value determined by the number of digits of the first identification data is much greater than the number of original data. Or the preset constraint condition can also be that the maximum value determined by the number of digits after the first identification data is reduced is greater than the number of original data. The preset conversion rule and the second identification data can be determined according to the preset constraint condition. The data type of the second identification data can be a short type, the data type of the second identification data can also be an Integer type, the data type of the second identification data can also be a string type, and the data type of the second identification data can be set according to actual needs. In this solution, if the first identification data and the second identification data are in the same base, the number of digits of the second identification data is less than the number of digits of the first identification data. If the first identification data and the second identification data are in different bases, the maximum value determined by the number of digits of the first identification data is greater than the maximum value determined by the number of digits of the second identification data.

[0033] In this solution, the original data corresponds to the second identification data one by one, and the number of bits of the second identification data is determined by the number of original data. For example, the number of national administrative divisions is less than 4,000, and the maximum value of a signed short integer in a computer is 32,767, so the data type of the second identification data can be set to short. Although the national standard takes into account "human readable and understandable" when defining dictionary encoding, for example, the national administrative division code defined in the national standard is expressed in 6 decimal digits, and the 6 digits are divided into three regions, namely: the first two digits represent provinces / autonomous regions / municipalities directly under the central government, the middle two digits represent cities, and the last three digits represent districts / counties, but this setting requires the use of Integer type or String type to represent and store enumeration fields. The maximum value determined by the number of bits of the Integer type or String type far exceeds the number of national administrative divisions. In the calculation process, data processing of the Integer type or String type takes up too much storage and computing resources.

[0034] The technical solution provided by the embodiment of the present invention obtains raw data of an enumeration type; determines first identification data according to the raw data; wherein the raw data is uniquely associated with the first identification data; if it is determined that the number of raw data and the number of bits of the first identification data meet the preset constraint condition, the raw data is converted into second identification data according to a preset conversion rule; wherein the raw data is uniquely associated with the second identification data, and the number of raw data matches the number of bits of the second identification data. By executing the technical solution provided by the embodiment of the present invention, it is possible to save the computing and storage space of the computer and improve the computing efficiency and retrieval efficiency.

[0035] Figure 2 1 is a flow chart of a data processing method provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiment. Figure 2 As shown, the data processing method in the embodiment of the present invention may include:

[0036] S210: Acquire original data whose data type is an enumeration type.

[0037] S220: Determine first identification data according to the original data.

[0038] S230: Determine the amount of the original data, and determine a value range according to the number of bits of the first identification data.

[0039] This solution can determine the number of original data and determine the value range according to the number of digits of the first identification data. For example, taking the original data as national administrative divisions as an example, the number of national administrative divisions can be determined, for example, less than 4000. Taking the first identification data citing the national standard as an example, the first identification data is the first six digits of the personal ID number, then the value range determined by the number of digits of the first identification data is 1 to 999999.

[0040] S240: If the amount of the original data is smaller than the value range of the first identification data after the number of bits is reduced, converting the original data into second identification data according to a preset conversion rule.

[0041] Among them, taking the original data as a national administrative division as an example, taking the first identification data citing the national standard as an example, the first identification data is the first six digits of the personal ID number. The value range of the first identification data after the number of digits is reduced by 1 is 1 to 99999. The value range of the first identification data after the number of digits is reduced by 2 is 1 to 9999. It can be determined that the number of original data is less than 99999 and 9999. This solution can convert the original data into the second identification data according to the preset conversion rule.

[0042] In a feasible implementation, optionally, if the amount of original data is smaller than the value range of the first identification data after the number of bits is reduced, the original data is converted into second identification data according to a preset conversion rule, including: if the amount of original data is smaller than the value range of the first identification data after the number of bits is reduced, determining the second data type according to the amount of original data; converting the original data into the second identification data according to the preset conversion rule and the second data type.

[0043] Exemplarily, taking the original data as a national administrative division as an example, taking the first identification data citing the national standard as an example, the first identification data is the first six digits of a personal ID number. The value range of the first identification data after the number of digits is reduced by 1 is 1 to 99999. The value range of the first identification data after the number of digits is reduced by 2 is 1 to 9999. It can be determined that the number of original data is less than 99999 and 9999. The number of original data can be represented and stored in four decimal digits. Each decimal digit can be represented by a 4-bit binary number, so the second data type can be determined as a short type based on the number of original data. This solution can convert the original data into a second identification data of short type according to the preset conversion rules and the second data type. Among them, the conversion rules between bases can refer to the relevant technology.

[0044] Thus, if the number of original data is less than the value range after the number of bits of the first identification data is reduced, the second data type is determined according to the number of original data; and the original data is converted into the second identification data according to the preset conversion rule and the second data type. It is possible to determine the appropriate data type according to the number of original data, and use the determined data type to represent and store enumerated type data, which can save computer storage resources and computing resources, and thus improve data processing efficiency when using the original data for data processing.

[0045] In another feasible implementation manner, optionally, after converting the original data into the second identification data according to a preset conversion rule, it further includes: establishing and storing a unique association relationship between the original data, the first identification data, and the second identification data.

[0046] Among them, the unique association relationship between the original data and the first identification data is predetermined. After the original data is converted into the second identification data according to the preset conversion rule, the unique association relationship between the original data and the second identification data can be determined. Then, through the unique association relationship between the original data and the first identification data and the unique association relationship between the original data and the second identification data, the unique association relationship between the original data, the first identification data and the second identification data can be determined and established, and the unique association relationship between the three can be stored.

[0047] Therefore, by establishing and storing a unique association between the original data, the first identification data and the second identification data, it is possible to efficiently and quickly achieve mutual conversion between the original data, the first identification data and the second identification data during the data processing process, and to obtain the required data according to actual needs, thereby improving the efficiency of data processing.

[0048] In this embodiment, optionally, after establishing and storing a unique association relationship between the original data, the first identification data, and the second identification data, the method further includes: responding to a first identification data acquisition request, determining a second identification data acquisition request based on the unique association relationship; determining the second identification data based on the second identification data acquisition request; and determining the first identification data based on the unique association relationship and the second identification data.

[0049] Wherein, after completing the conversion between the original data and the second identification data, the obtained second identification data is sent to the target subsystem for storage, so that the target subsystem receives the first identification data acquisition request from the external system, and then returns the first identification data to the external system according to the first identification data acquisition request. Wherein, the external system can be a system that sends a first identification data operation request, such as a first identification data query request. The specific implementation process is that, according to the first identification data acquisition request and the unique association relationship between the original data, the first identification data and the second identification data, the first identification data in the first identification data acquisition request can be converted into the second identification data, and then the second identification data acquisition request is obtained. According to the second identification data acquisition request, the data operation matching the second identification data acquisition request is performed in the target subsystem to obtain the second identification data. The target subsystem determines the first identification data matching the first identification data acquisition request according to the obtained second identification data and the unique association relationship between the original data, the first identification data and the second identification data, and feeds it back to the external system.

[0050] Thus, by responding to the first identification data acquisition request, determining the second identification data acquisition request according to the unique association relationship; determining the second identification data according to the second identification data acquisition request; and determining the first identification data according to the unique association relationship and the second identification data, the second identification data and the second identification data can be efficiently and quickly converted to each other during the data operation process, the required data can be obtained according to actual needs, and the efficiency of data retrieval and data processing can be improved.

[0051] The technical solution provided by the embodiment of the present invention obtains raw data of an enumeration type; determines first identification data according to the raw data; determines the number of raw data, and determines a value range according to the number of bits of the first identification data; if the number of raw data is less than the value range after the number of bits of the first identification data is reduced, the raw data is converted into second identification data according to a preset conversion rule. By executing this solution, it is possible to save the computing and storage space of the computer and improve the computing efficiency and retrieval efficiency.

[0052] Figure 3a It is a schematic diagram of the data processing system structure of the prior art, such as Figure 3a As shown, the original data is extracted from the source subsystem, and the extracted data is loaded into the target subsystem by the ETL subsystem after conversion. During the ETL conversion process, the dictionary field is converted into a format, and the dictionary code of the dictionary field is converted from the original data to the first identification data defined by the target subsystem. When the external system consumes the data of the target subsystem, it uses the first identification data to interact with the target subsystem. The mapping relationship between the original data and the first identification data is initialized into the ETL subsystem by the maintenance personnel.

[0053] In the big data ETL processing process of related technologies, the value of the processed data field is an enumeration type called an enumeration field, also known as a dictionary field. The value range of the dictionary field is generally defined to include a dictionary code, a dictionary value, and the corresponding description information, where the dictionary code is generally defined as a numeric type (decimal or hexadecimal), and the dictionary value is usually a text expression. In addition to being easy to process, the dictionary code is generally defined to be "human readable and understandable". For example, the national administrative division code defined in the national standard is expressed in 6 decimal digits, and the 6 digits are divided into three regions, namely: the first two digits represent the province / autonomous region / municipality, the middle two digits represent the city, and the last three digits represent the district / county. For example, 659001 represents a certain province, a certain city, and a certain county, among which 659001 belongs to the dictionary code, and "a certain province, a certain city, and a certain county" belongs to the dictionary value. Although 659001 is a string of decimal numbers, it exceeds the maximum expression range of 32767 of Java's signed short integer (short, occupies 2 bytes). It needs to be expressed using the Integer type or the String type. The Integer type occupies 4 bytes, and the String type occupies 6 bytes.

[0054] In the technical solution of the prior art, the first identification data output by the ETL subsystem to the target subsystem generally has business significance and has the characteristic of being "human-readable", which results in the first identification data being relatively long. The longer the first identification data and the more dictionary fields there are, the more computing and storage resources are required, especially in the context of big data.

[0055] According to statistics, the enumeration value range of administrative divisions is more than 3500. If an integer number is assigned to each administrative division code, Java's signed short integer (short, occupies 2 bytes) is sufficient, which can save 2 bytes compared to the Integer type and 4 bytes compared to the String type.

[0056] In the context of big data, the daily data increment in the system is hundreds of millions, which requires a large amount of computing and processing resources and storage and retrieval resources to support. Assuming that the daily increment of the system is 1 billion, according to the above example of administrative division codes, using unsigned short integers can save 2GB of space compared to the Integer method and 4GB of space compared to the String method. Furthermore, the total number of enumeration fields in the system is more than one. The more enumeration fields there are, the more space there is for optimization; in addition, the dictionary code length of each enumeration field may be more than 6 digits. The longer the dictionary code length of the enumeration field, the more space there is for optimization. Using a data type that occupies less space to store the dictionary codes of all enumeration fields in the system will effectively reduce the required computing and processing resources and storage and retrieval resources.

[0057] In order to better illustrate the technical solution, some concepts need to be defined and explained: original data, first identification data and second identification data. Original data refers to the dictionary encoding of enumeration class fields in the source subsystem, i.e., the input of enumeration class fields in the ETL subsystem; first identification data refers to the dictionary encoding of enumeration class fields in the target subsystem, which generally refers to some standards such as national standards, industry standards, enterprise standards, etc. For example, the national administrative division code generally refers to the above national standards, i.e., the output of enumeration class fields in the ETL subsystem; second identification data refers to a data type generated by an embodiment of the present invention that is shorter than the original data and the first identification data. The second identification data replaces the first identification data for calculation, processing and storage indexing in the target subsystem.

[0058] Figure 3b Schematic diagram of the structure of a data processing system applicable to an embodiment of the present invention. In order to more clearly describe the technical solution of the present invention, Figure 3b As shown, the technical solution provided by the embodiment of the present invention may include the following subsystems:

[0059] On the basis of the data processing system of the prior art, the embodiment of the present invention adds a second identification subsystem. The second identification subsystem maintains the mapping relationship between the original data, the second identification data, and the first identification data, and interacts with the ETL subsystem and the target subsystem to achieve the purpose of saving resources. Among them, the mapping relationship between the original data and the first identification data is initialized into the second identification subsystem by the maintenance personnel. During the initialization process, the second identification subsystem will generate the second identification data corresponding to the corresponding original data and the first identification data.

[0060] Before further data processing, the ETL subsystem first inputs the original data received from the source subsystem into the second identification subsystem to obtain the corresponding second identification data. The subsequent processing of the ETL subsystem is based on the second identification data. The subsequent processing based on the second identification data will reduce the computing resources and storage resources required in the processing process. The ETL subsystem outputs the processed data containing the second identification data to the target subsystem. The target subsystem can reduce the corresponding resource consumption by using the second identification data to store and process data.

[0061] The external system uses the first identification data to consume data with the target subsystem. After receiving the first identification data acquisition request, the target subsystem sends the first identification data to the second identification subsystem. The second identification subsystem returns the corresponding second identification data to the target subsystem. The target subsystem uses the second identification data to perform corresponding data processing, converts the processed result into the first identification data, and then feeds it back to the external system. The second identification data is imperceptible to the external system during the entire data consumption process.

[0062] The technical solution provided by the embodiment of the present invention can reduce the space occupied by dictionary fields under the same data scale, thereby effectively reducing the required computing and processing resources and storage and retrieval resources. Based on the above capabilities, data producers will use fewer computing and storage resources when computing, processing, storing and indexing data of the same scale, and provide data access capabilities on this basis. Data consumers' access to data services will not be affected in any way.

[0063] Figure 4 FIG. 1 is a schematic diagram of a data processing device provided by an embodiment of the present invention. The device may be configured in an electronic device for data processing. Figure 4 As shown, the device comprises:

[0064] The original data acquisition module 410 is used to acquire original data whose data type is an enumeration type;

[0065] A first identification data determination module 420, configured to determine first identification data according to the original data; wherein the original data is uniquely associated with the first identification data;

[0066] The data conversion module 430 is used to convert the original data into second identification data according to a preset conversion rule if it is determined that the amount of original data and the number of bits of the first identification data meet the preset constraints; wherein the original data is uniquely associated with the second identification data, and the amount of original data matches the number of bits of the second identification data.

[0067] Optionally, the data conversion module 430 includes a value range determination unit, which is used to determine the amount of original data and determine the value range based on the number of bits of the first identification data; a data conversion unit, which is used to convert the original data into second identification data according to a preset conversion rule if the amount of original data is smaller than the value range after the number of bits of the first identification data is reduced.

[0068] Optionally, the data conversion unit includes a second data type determination subunit, which is used to determine the second data type based on the number of original data if the number of original data is smaller than the value range of the first identification data after the number of bits is reduced; and the data conversion subunit is used to convert the original data into second identification data according to a preset conversion rule and the second data type.

[0069] Optionally, the device further includes an association relationship storage module, which is used to establish and store a unique association relationship between the original data, the first identification data and the second identification data after converting the original data into the second identification data according to a preset conversion rule.

[0070] Optionally, the method also includes a request response module, which is used to respond to the first identification data acquisition request and determine the second identification data acquisition request based on the unique association relationship; determine the second identification data based on the second identification data acquisition request; and determine the first identification data based on the unique association relationship and the second identification data.

[0071] The device provided in the above embodiments can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0072] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the device includes:

[0073] one or more processors 510, Figure 5 A processor 510 is taken as an example;

[0074] Memory 520;

[0075] The device may further include: an input device 530 and an output device 540 .

[0076] The processor 510, memory 520, input device 530 and output device 540 in the device may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0077] The memory 520, as a non-transitory computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to a data processing method in an embodiment of the present invention. The processor 510 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 520, that is, a data processing method of the above method embodiment is implemented, namely:

[0078] Get the original data whose data type is enumeration type;

[0079] Determining first identification data according to the original data; wherein the original data is uniquely associated with the first identification data;

[0080] If it is determined that the amount of original data and the number of bits of the first identification data meet the preset constraints, the original data is converted into second identification data according to a preset conversion rule; wherein the original data is uniquely associated with the second identification data, and the amount of original data matches the number of bits of the second identification data.

[0081] The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 520 may optionally include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] The input device 530 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 540 may include a display device such as a display screen.

[0083] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a data processing method provided by an embodiment of the present invention is implemented, that is,

[0084] Get the original data whose data type is enumeration type;

[0085] Determining first identification data according to the original data; wherein the original data is uniquely associated with the first identification data;

[0086] If it is determined that the amount of original data and the number of bits of the first identification data meet the preset constraints, the original data is converted into second identification data according to a preset conversion rule; wherein the original data is uniquely associated with the second identification data, and the amount of original data matches the number of bits of the second identification data.

[0087] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0088] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0089] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0090] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: Get the original data whose data type is enumeration type; Determining first identification data according to the original data; wherein the original data is uniquely associated with the first identification data; If it is determined that the amount of the original data and the number of bits of the first identification data meet the preset constraint condition, converting the original data into second identification data according to a preset conversion rule; wherein the original data is uniquely associated with the second identification data, and the amount of the original data matches the number of bits of the second identification data; Wherein, the first identification data is the output of the enumeration class field type data in the ETL processing process; the ETL processing process is the process of loading the original data into the target subsystem after extraction, cleaning and conversion; If it is determined that the amount of the original data and the number of bits of the first identification data meet a preset constraint condition, converting the original data into second identification data according to a preset conversion rule includes: Determine the amount of the original data, and determine a value range according to the number of bits of the first identification data; If the amount of the original data is smaller than the value range of the first identification data after the number of bits is reduced, the original data is converted into second identification data according to a preset conversion rule.

2. The method according to claim 1, characterized in that If the amount of the original data is smaller than the value range of the first identification data after the number of bits is reduced, converting the original data into second identification data according to a preset conversion rule includes: If the amount of the original data is smaller than the value range of the first identification data after the number of bits is reduced, determining the second data type according to the amount of the original data; The original data is converted into second identification data according to a preset conversion rule and the second data type.

3. The method according to claim 1, characterized in that: After converting the original data into the second identification data according to the preset conversion rule, the method further includes: A unique association relationship between the original data, the first identification data, and the second identification data is established and stored.

4. The method according to claim 3, characterized in that The method further comprises: In response to the first identification data acquisition request, determining a second identification data acquisition request according to the unique association relationship; determining the second identification data according to the second identification data acquisition request; The first identification data is determined according to the unique association relationship and the second identification data.

5. A data processing device, characterized in that: include: The original data acquisition module is used to acquire the original data whose data type is an enumeration type; A first identification data determination module, configured to determine first identification data according to the original data; wherein the original data is uniquely associated with the first identification data; A data conversion module, configured to convert the original data into second identification data according to a preset conversion rule if it is determined that the original data quantity and the number of bits of the first identification data meet a preset constraint condition; wherein the original data are uniquely associated with the second identification data, and the original data quantity matches the number of bits of the second identification data; Wherein, the first identification data is the output of the enumeration class field type data in the ETL processing process; the ETL processing process is the process of loading the original data into the target subsystem after extraction, cleaning and conversion; The data conversion module includes a value range determination unit, which is used to determine the amount of the original data and determine the value range according to the number of bits of the first identification data; the data conversion unit is used to convert the original data into second identification data according to a preset conversion rule if the amount of the original data is smaller than the value range after the number of bits of the first identification data is reduced.

6. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images