A data processing method, electronic device and computer readable storage medium

Abstract

The present application discloses a data processing method, an electronic device and a computer-readable storage medium. The method comprises: obtaining data to be stored from a first data storage module according to an acquired data storage instruction, wherein the data to be stored includes an initial value to be stored and a field to be stored corresponding to the initial value; storing the initial value in a preset array to obtain an initial array, wherein the initial array includes initial values ​​with different values; constructing a target data structure according to the initial array and the fields to be stored corresponding to each initial value; and retrieving the target value from the target data structure and writing it into the second data storage module according to the data storage rule of the second data storage module. Thus, the storage space occupied by the data can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the development of digital technology, data is becoming more and more complex. In order to facilitate access and analysis, the data integration platform integrates data from multiple different data sources into a unified, easy-to-access and easy-to-analyze data repository. The process of data integration is to process the data from each data source and transfer it to another data repository.

[0003] At present, the processing method for transferring stored data is to retrieve all the data from a data source in the system and then write all the retrieved data into another data storage repository. However, this method requires a large amount of storage space in the system to store the retrieved data. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a data processing method, an electronic device and a computer-readable storage medium, which can reduce the storage space occupied by data.

[0005] In order to solve the above technical problems, the present application provides a data processing method, an electronic device and a computer-readable storage medium.

[0006] In one embodiment, a data processing method is applied to a data processing system, the data processing system includes a data processing module, a first data storage module and a second data storage module, one end of the data processing module is connected to the first data storage module, and the other end is connected to the second data storage module, the method includes: the data processing module obtains the data to be stored from the first data storage module according to the acquired data storage instruction, the data to be stored includes an initial value to be stored and a field to be stored corresponding to the initial value; the initial value is stored in a preset array to obtain an initial array, the initial array includes initial values ​​with different values; a target data structure is constructed according to the initial array and the fields to be stored corresponding to each initial value; according to the data storage rule of the second data storage module, the target value is retrieved from the target data structure and written into the second data storage module.

[0007] In one embodiment, the step of storing the initial numerical value in a preset array to obtain the initial array includes: determining whether there is a numerical value in the preset array that is the same as the initial numerical value to be stored; if not, storing the initial numerical value in the preset array to obtain the initial array; if so, using the same numerical value as the initial numerical value stored in the preset array, and using the storage position of the same numerical value in the preset array as the storage position of the initial numerical value in the preset array to obtain the initial array.

[0008] In one embodiment, the step of constructing a target data structure based on the initial array and the fields to be stored corresponding to each initial value includes: determining a field offset based on the storage position of the initial value in the initial array; constructing an initial table structure corresponding to the field to be stored based on the fields to be stored corresponding to the initial value and the field offset corresponding to the initial value; performing rule change processing on the initial table structure and / or initial array according to preset data transformation rules to obtain a target data structure.

[0009] In one embodiment, the preset data transformation rules include at least one of field renaming rules, field modification rules, field addition rules and field deletion rules. The step of performing rule change processing on the initial table structure and / or initial array according to the preset data transformation rules to obtain the target data structure includes: renaming the fields in the initial table structure according to the field renaming rules to obtain a first target table structure; and / or, changing the initial values ​​in the initial array according to the field modification rules to obtain the target values; and / or, deleting the fields in the initial table structure according to the field deletion rules to obtain a second target table structure; and / or, adding the fields in the initial table structure according to the field addition rules to obtain a third target table structure; determining the target data structure according to the first target table structure, the second target table structure, the third target table structure and / or the target values.

[0010] In one embodiment, the field renaming rule includes a first field name and a target field name corresponding to the first field name, and the field in the initial table structure is renamed according to the field renaming rule to obtain the first target table structure, including: matching the field name to be replaced from the initial table structure according to the first field name; replacing the field name to be replaced in the initial table structure with the corresponding target field name to obtain the first target table structure.

[0011] In one embodiment, the field modification rule includes a second field name and an algorithm corresponding to the second field name, and the step of changing the initial value in the initial array according to the field modification rule to obtain the target value includes: matching the field name to be processed from the initial table structure according to the second field name; determining the initial value to be modified from the initial array according to the field offset corresponding to the field name to be processed in the initial table structure; and calculating the initial value to be modified according to the algorithm corresponding to the second field name to obtain the target value.

[0012] In one embodiment, the step of retrieving a target value from the target data structure and writing it into the second data storage module according to the data storage rule of the second data storage module includes: obtaining the data to be written according to the target value retrieved from the target data structure; in response to the target storage format of the second data storage module being inconsistent with the format of the data to be written, performing format conversion on the data to be written according to the target storage format to obtain converted data; and writing the converted data into the second data storage module.

[0013] In one embodiment, the target data structure includes a storage position of a target table structure in the data processing module and a target array, the target array includes target values ​​with different values, and the step of obtaining the data to be written according to the target value retrieved from the target data structure includes: determining a target storage field and a field offset corresponding to the target storage field from the target table structure according to the storage position of the target table structure in the data processing module; retrieving the target value from the target array according to the field offset; and performing format conversion processing on the target value according to the target storage field to obtain the data to be written.

[0014] In order to solve the above technical problems, the present application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above data processing method.

[0015] In order to solve the above technical problems, the present application provides a computer-readable storage medium, including: storing program data, and the program data is used to implement the above data processing method when executed by a processor.

[0016] The above scheme obtains the data to be stored from the first data storage module according to the acquired data storage instruction, wherein the data to be stored includes the initial value to be stored and the field to be stored corresponding to the initial value; the initial value is stored in a preset array to obtain an initial array, wherein the initial array includes initial values ​​with different values; a target data structure is constructed according to the initial array and the fields to be stored corresponding to each initial value; and the target value is retrieved from the target data structure and written into the second data storage module according to the data storage rule of the second data storage module. On the one hand, the target value is called from the target data structure constructed from the initial array and the field to be stored, and then written into the second data storage module, thereby realizing the transfer of stored data. On the other hand, using the initial array to store initial values ​​with different values ​​can avoid repeated storage of the same value, reduce the storage space of data, and thus realize the reduction of data storage space in the process of transferring stored data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0018] Figure 1 It is a structural schematic diagram of an exemplary embodiment of a data processing method shown in the present application;

[0019] Figure 2 yes Figure 1 A flow chart of an exemplary embodiment of step S120 in the data processing method is shown;

[0020] Figure 3 yes Figure 1 A flow chart of an exemplary embodiment of step S130 in the data processing method is shown;

[0021] Figure 4 yes Figure 3 A flow chart of an exemplary embodiment of step S330 in the data processing method is shown;

[0022] Figure 5 yes Figure 3 FIG. 1 is a flow chart of another exemplary embodiment of step S330 in the data processing method shown;

[0023] Figure 6 It is a schematic diagram of the process of the operator processing the initial table structure shown in this application;

[0024] Figure 7is a block diagram of a data processing device shown in an exemplary embodiment of the present application;

[0025] Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0026] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0028] First of all, it should be noted that with the development of digital technology, data is becoming more and more complex. In order to facilitate access and analysis, the data integration platform integrates data from multiple different data sources into a unified, easy-to-access and easy-to-analyze data repository. The process of data integration is to process the data from each data source and transfer it to another data repository.

[0029] At present, the processing method for transferring stored data is to retrieve all the data from a data source in the system and then write all the retrieved data into another data storage repository. However, this method requires a large amount of storage space in the system to store the retrieved data.

[0030] Based on this, the present application provides a data processing method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1 It is a structural diagram of an exemplary embodiment of a data processing method shown in the present application.

[0031] The execution subject of a data processing method may be a terminal device or a server or other processing device, wherein the terminal device may be a computer, a mobile device, a terminal, a computing device, a vehicle-mounted device, etc. The execution subject of the data processing method may also be a data processing device. In some possible implementations, the data processing method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0032] The data processing method is applied to a data processing system, and the data processing system includes a data processing module, a first data storage module and a second data storage module, one end of the data processing module is connected to the first data storage module, and the other end is connected to the second data storage module. Among them, the first data storage module stores original data, the data processing module exists in the data processing device, and the second data storage module is used to store data obtained by the data processing method of the present application. As an example, the original data can be structured data, such as data stored in relational databases such as MySQL (MySQL Database Server), PostgreSQL (PostgreSQL Database Management System), Oracle (Oracle Database) and SQL Server (Microsoft SQL Server). The original data can also be semi-structured data, such as XML (eXtensible Markup Language, a markup language), JSON (JavaScript Object Notation, a lightweight data exchange format) and CSV (Comma-Separated Values, a simple file format) files, etc.

[0033] Specifically, a data processing method provided in this embodiment includes the following steps:

[0034] Step S110: The data processing module obtains the data to be stored from the first data storage module according to the obtained data storage instruction, where the data to be stored includes the initial value to be stored and the field to be stored corresponding to the initial value.

[0035] The data storage instruction is an instruction to store data. The data storage instruction may be an instruction to store data in the first data storage module into the second data storage module. As an example, the data processing module obtains the input storage instruction and uses the input storage instruction as the data storage instruction.

[0036] The data to be stored are the values ​​and fields waiting to be stored. The data to be stored include the initial values ​​to be stored and the fields to be stored corresponding to the initial values. The initial values ​​are the numbers to be stored in the first data storage module. For example, the initial values ​​can be 1730183367, 1, 1, 2024-0101, 1, and 1. The fields to be stored are the field information of the initial values ​​in the first data storage module. For example, the fields to be stored can include the field name, and can also include one or more of the type, length, precision, and time format. The data to be stored can be a key value and a value value associated with the key, wherein the key value is the field to be stored, and the value value is the initial value corresponding to the key value.

[0037] The data processing module obtains the data to be stored from the first data storage module according to the acquired data storage instruction. Specifically, in response to receiving the data storage instruction, the data processing module parses the original data in the first data storage module, obtains the initial value to be stored and the field to be stored corresponding to the initial value, and uses the initial value to be stored and the field to be stored corresponding to the initial value as the data to be stored. As an example, the data processing module uses a data operation language to query structured data to obtain the initial value to be stored and the field to be stored corresponding to the initial value, and the data operation language can be SQL language. As another example, the data processing module uses a non-relational database language to query semi-structured data to obtain the initial value to be stored and the field to be stored corresponding to the initial value. The non-relational database language can be MongoDB language, Couchbase language, etc.

[0038] Step S120: storing the initial values ​​into a preset array to obtain an initial array, wherein the initial array includes initial values ​​with different values.

[0039] A preset array refers to an array used to store numerical values ​​in a data processing module.

[0040] The initial array refers to an array of initial values ​​with different values ​​stored in the data processing module. For example, the initial array can be [1, 2024-0101].

[0041] The data processing module stores the initial values ​​in a preset array to obtain an initial array. Specifically, there is no stored value in the preset array, and the data processing module selects initial values ​​with different values ​​from the initial values, and stores the selected initial values ​​in sequence in the preset array to obtain an initial array.

[0042] Step S130: construct a target data structure according to the initial array and the fields to be stored corresponding to each initial value.

[0043] The target data structure is a structure for storing data. The target data structure may include a target array, a target table structure, or a target array and a target table structure pointer. The target table structure pointer represents the storage position of the target table structure in the data processing module. The target array is an array storing target values ​​with different values. The target value may be a value obtained after calculation of the initial value in the initial array, or may be the initial value in the initial array. The target table structure is a table storing field offsets and fields to be stored. The field offset represents the storage position of the target value in the target array, or the storage position of the initial value in the initial array.

[0044] The data processing module constructs a target data structure according to the initial array and the fields to be stored corresponding to each initial value. As an example, the data processing module uses the number of the storage position of each initial value in the initial array as the field offset; stores the field offset and the field to be stored corresponding to each initial value in the preset table structure to obtain the target table structure, and uses the target table structure as the target data structure. As another example, the data processing module uses the number of the storage position of each initial value in the initial array as the field offset; stores the field offset and the field to be stored corresponding to each initial value in the preset table structure to obtain the target table structure; uses the initial array as the target array, the initial values ​​in the initial array as the target values ​​in the target array, and uses the storage position of the target table structure in the data processing module and the target array as the target data structure.

[0045] Step S140: Retrieving a target value from the target data structure and writing it into the second data storage module according to the data storage rule of the second data storage module.

[0046] The data storage rule refers to a storage method that enables data to be compatible with the second data storage module. The data storage rule may be a format, type, etc. of data storage.

[0047] The data processing module retrieves the target value from the target data structure and writes it into the second data storage module according to the data storage rule of the second data storage module. Specifically, the data processing module converts the target value retrieved from the target data structure according to the data storage rule, obtains the converted data, and writes the converted data into the second data storage module. For example, the data processing module performs type conversion on the target value retrieved from the target data structure according to the data storage type of the second data storage module, obtains the type-converted data, and stores the type-converted data into the second data storage module.

[0048] It can be seen that, according to the acquired data storage instruction, the data to be stored is obtained from the first data storage module, and the data to be stored includes the initial value to be stored and the field to be stored corresponding to the initial value; the initial value is stored in the preset array to obtain the initial array, and the initial array includes initial values ​​with different values; the target data structure is constructed according to the initial array and the fields to be stored corresponding to each initial value; according to the data storage rule of the second data storage module, the target value is retrieved from the target data structure and written into the second data storage module. On the one hand, the target value is called from the target data structure constructed by the initial array and the field to be stored, and then written into the second data storage module, so as to realize the transfer of the stored data. On the other hand, the use of the initial array to store initial values ​​with different values ​​can avoid the repeated storage of the same value, reduce the storage space of the data, and thus realize the reduction of the storage space of the data in the process of transferring the stored data.

[0049] Based on the above examples, please refer to Figure 2 , Figure 2 yes Figure 1 The flowchart of an exemplary embodiment of step S120 in the data processing method is shown. Specifically, step S120 stores the initial value into a preset array, and the process of obtaining the initial array includes the following steps:

[0050] Step S210, determine whether there is a value in the preset array that is the same as the initial value to be stored, if not, execute step S220, if yes, execute step S230.

[0051] The data processing module determines whether there is a value identical to the initial value to be stored in the preset array. Specifically, the data processing module searches for each initial value to be stored in the preset array in turn. If a value is found, it is determined that there is a value identical to the initial value to be stored in the preset array, and the found value is determined to be the same value; if no value is found, it is determined that there is no value identical to the initial value to be stored in the preset array.

[0052] Step S220: store the initial values ​​into a preset array to obtain an initial array.

[0053] The data processing module determines that there is no value in the preset array that is the same as the initial value to be stored, and stores the initial value in the preset array to obtain the initial array.

[0054] Step S230: Using the same value as the initial value stored in the preset array, and using the storage position of the same value in the preset array as the storage position of the initial value in the preset array to obtain an initial array.

[0055] Specifically, if the data processing module determines that there is a value in the preset array that is the same as the initial value to be stored, the initial value will not be stored, the same value in the preset array will be used as the initial value stored in the preset array, and the storage position of the same value in the preset array will be used as the storage position of the initial value in the preset array to obtain the initial array.

[0056] In one embodiment, the data processing module traverses the initial values ​​to be stored, and determines in turn whether the current initial values ​​to be stored have the same value in the preset array; if not, the current initial value is stored in the preset array; if so, the same value is used as the initial value stored in the preset array, and the storage position of the same value in the preset array is used as the storage position of the current initial value in the preset array, until all the initial values ​​to be stored are stored in the preset array to obtain the initial array. In this way, all initial values ​​with different values ​​are stored in the data processing module, the storage of repeated values ​​is reduced, and thus the storage space of data is reduced.

[0057] Based on the above examples, please refer to Figure 3 , Figure 3 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the data processing method is shown. Specifically, the process of step S130 constructing the target data structure according to the initial array and the fields to be stored corresponding to each initial value includes the following steps:

[0058] Step S310: determine the field offset according to the storage position of the initial value in the initial array.

[0059] The field offset represents the number of the storage location in the initial array. For example, the field offset may be 0, representing the storage location numbered 0 in the initial array.

[0060] The data processing module determines the field offset according to the storage position of the initial value in the initial array. As an example, the data processing module determines the number corresponding to the storage position of the initial value in the initial array as the field offset corresponding to the initial value. As another example, the data storage module determines the number corresponding to the storage position of the same value in the initial array as the field offset corresponding to the initial value. Thus, by characterizing the storage position of the initial value in the initial array through the field offset, the initial value is associated with the field offset, which can reduce the storage of the associated key value of the initial value, and under the logic of normal data flow, a large amount of data key value memory usage can be reduced.

[0061] Step S320: construct an initial table structure corresponding to the field to be stored according to the field to be stored corresponding to the initial value and the field offset corresponding to the initial value.

[0062] The initial table structure is a table storing field information. The initial table structure may include field offsets and fields to be stored. For example, the initial table structure may include field name A, field type int, and field offset 0. It should be noted that the initial table structure may be stored in an operator of a data processing module.

[0063] The data processing module constructs an initial table structure corresponding to the field to be stored according to the field to be stored corresponding to the initial value and the field offset corresponding to the initial value. Specifically, the data processing module stores the field to be stored and the field offset corresponding to at least one initial value into a preset table structure to obtain an initial table structure. The preset table structure is a table that does not store information.

[0064] Step S330: Perform rule transformation processing on the initial table structure and / or the initial array according to preset data transformation rules to obtain a target data structure.

[0065] The preset data transformation rule refers to a method for transforming a field or a value. The preset data transformation rule may include at least one of a field renaming rule, a field modification rule, a field addition rule, and a field deletion rule. As an example, the data processing module obtains the rule information input by the object, performs text parsing on the rule information input by the object, and obtains the preset data transformation rule. As another example, the data processing module retrieves the data transformation rule in the preset transformation rule database, and uses the retrieved data transformation rule as the preset data transformation rule.

[0066] Specifically, the data processing module performs rule change processing on the initial table structure and / or the initial array according to the preset data transformation rule to obtain the step of the target data structure, including: renaming the fields in the initial table structure according to the field renaming rule to obtain the first target table structure; and / or, changing the initial values ​​in the initial array according to the field modification rule to obtain the target value; and / or, deleting the fields in the initial table structure according to the field deletion rule to obtain the second target table structure; and / or, adding the fields in the initial table structure according to the field addition rule to obtain the third target table structure; determining the target data structure according to the first target table structure, the second target table structure, the third target table structure and / or the target value. Thus, by constructing the table structure and the array, when performing data transformation for renaming, deleting, and adding a large number of fields, it is only necessary to modify the content in the table structure, and it is not necessary to modify the values ​​in the array, which improves the efficiency and convenience of data transformation, and can reduce the CPU (Central Processing Unit) consumption caused by data conversion calculation.

[0067] Based on the above examples, please refer to Figure 4 , Figure 4 yes Figure 3 The flowchart of an exemplary embodiment of step S330 in the data processing method is shown. Specifically, step S330 performs rule transformation processing on the initial table structure and / or initial array according to the preset data transformation rule, and the process of obtaining the target data structure includes the following steps:

[0068] Step S410: rename the fields in the initial table structure according to the field renaming rule to obtain a first target table structure.

[0069] The field renaming rule refers to a method for renaming a field. The field renaming rule may include a first field name and a target field name corresponding to the first field name.

[0070] The data processing module renames the fields in the initial table structure according to the field renaming rule to obtain a first target table structure. Specifically, the data processing module matches the field name to be replaced from the initial table structure according to the first field name; replaces the field name to be replaced in the initial table structure with the corresponding target field name to obtain the first target table structure. For example, the first field name in the field renaming rule is A and the target field name corresponding to the first field name A is E. The data processing module matches the field name to be replaced A from the initial table structure, replaces A with E, and obtains the first target table structure.

[0071] Step S420: Change the initial values ​​in the initial array according to the field modification rule to obtain the target values.

[0072] The field modification rule refers to a method for modifying the value corresponding to the field. The field modification rule may include a second field name and an algorithm corresponding to the second field name. The corresponding algorithm may be adding a first preset threshold value, or subtracting a second preset threshold value, etc.

[0073] The data processing module changes the initial value in the initial array according to the field modification rule to obtain the target value. Specifically, the data processing module matches the field name to be processed from the initial table structure according to the second field name; determines the initial value to be modified from the initial array according to the field offset corresponding to the field name to be processed in the initial table structure; calculates the initial value to be modified according to the algorithm corresponding to the second field name to obtain the target value. For example, the second field name in the field modification rule is D and the algorithm corresponding to the second field name is plus 1, the data processing module matches the field name to be processed D from the initial table structure according to the second field name D, determines that the field offset corresponding to the field name to be processed D is 3 from the initial table structure, finds out from the initial array that the initial value stored in the storage position numbered 3 is 8, adds 1 to 8, and the sum is 9, the sum is the target value, and the storage position of the sum in the initial array is the storage position of the target value.

[0074] Step S430: Delete the fields in the initial table structure according to the field deletion rule to obtain a second target table structure.

[0075] The field deletion rule refers to the method of deleting a field. The field deletion rule may include a third field name.

[0076] The data processing module deletes the fields in the initial table structure according to the field deletion rule to obtain the second target table structure. Specifically, the data processing module matches the field to be deleted from the initial table structure according to the third field name; and deletes the field to be deleted in the initial table structure; the field to be deleted includes the field name, other information corresponding to the field name, and the corresponding field offset, and the other information is the information in the field to be stored except the field name, to obtain the second target table structure. For example, the third field name in the field deletion rule is C, and the data processing module finds the field to be deleted with the field name C from the initial table structure; the field to be deleted includes the field name C, the field type int corresponding to the field name C, and the corresponding field offset is 2; the field to be deleted is deleted to obtain the second target table structure.

[0077] Step S440: Add fields in the initial table structure according to the field adding rule to obtain a third target table structure.

[0078] The field adding rule refers to the method of adding a field. The field adding rule may include the field to be added and the initial value corresponding to the field to be added.

[0079] The data processing module performs a new addition process on the fields in the initial table structure according to the field addition rule to obtain a third target table structure. Specifically, the data processing module matches the same initial value from the initial array according to the initial value corresponding to the field to be added; the storage position of the matched initial value in the initial array is used as the storage position of the initial value corresponding to the field to be added; the number of the storage position of the initial value corresponding to the field to be added is determined as the corresponding field offset; the field to be added and the field offset corresponding to the field to be added are stored in the initial table structure to obtain the third target table structure. For example, the field name of the field to be added in the field addition rule is D, the field type is int, and the initial value corresponding to the field to be added is 9. The data processing module matches the storage position number of 9 in the initial array from the initial array as 3, determines the number 3 as the field offset corresponding to the field to be added, stores the field name D, the field type int and the number 3 in the initial table structure to obtain the third target table structure.

[0080] It should be noted that the above steps S410 to S440 can be arbitrarily exchanged in order and can be arbitrarily combined. Specifically, the process of the data processing module obtaining the target data structure can include only any one of steps S410 to S440, for example, including step S420; the process of obtaining the target data structure can include any two of steps S410 to S440, for example, including step S410 and step S430; the process of obtaining the target data structure can also include any three of steps S410 to S440, for example, only including step S410, step S430 and step S440.

[0081] Step S450: determine a target data structure according to the first target table structure, the second target table structure, the third target table structure and / or the target value.

[0082] In one embodiment, the data processing module determines the target data structure according to the first target table structure, the second target table structure or the third target table structure. As an example, the data processing module uses the first target table structure, the second target table structure or the third target table structure as the target data structure. As another example, the data processing module uses the storage position of the first target table structure in the data processing module, the storage position of the second target table structure in the data processing module or the storage position of the third target table structure in the data processing module as the target data structure. As another example, the data processing module uses the storage position of the first target table structure in the data processing module, the storage position of the second target table structure in the data processing module or the storage position of the third target table structure in the data processing module, and the target array as the target data structure.

[0083] In another embodiment, the data processing module determines the target data structure according to the target value. As an example, the data processing module replaces the initial value to be modified in the initial array with the target value, stores other initial values ​​in the initial array as other target values ​​in the initial array, and obtains the target array; the storage position of the target table structure in the data processing module and the target array are determined as the target data structure. As another example, the data processing module stores the target value in the initial array, stores the initial values ​​in the initial array as other target values ​​in the initial array, and obtains the target array; the storage position of the target table structure in the data processing module and the target array are determined as the target data structure.

[0084] In another embodiment, the data processing module determines the target data structure according to the first target table structure and the second target table structure. As an example, the data processing module uses the storage position of the first target table structure in the data processing module and the storage position of the second target table structure in the data processing module as the target data structure. As another example, the data processing module merges the first target table structure and the second target table structure to obtain a merged target table structure; uses the initial array as the target array, and uses the initial values ​​in the initial array as the target values ​​in the target array; uses the storage position of the merged target table structure in the data processing module and the target array as the target data structure. The merging process includes: merging the fields to be stored in the first target table structure and the second target table structure and the field offsets corresponding to the fields to be stored into one target table structure, and deleting duplicate fields to be stored and their corresponding field offsets.

[0085] In another embodiment, the data processing module performs rule change processing on the initial table structure and / or the initial array according to the preset data transformation rules to obtain the target data structure. As an example, the data processing module performs renaming processing on the fields in the initial table structure according to the field renaming rule to obtain the first target table structure; and determines the target data structure according to the first target table structure. As another example, the data processing module performs renaming processing on the fields in the initial table structure according to the field renaming rule to obtain the first target table structure; performs change processing on the initial values ​​in the initial array according to the field modification rule to obtain the target values; and determines the target data structure according to the first target table structure and the target values. As another example, the data processing module performs renaming processing on the fields in the initial table structure according to the field renaming rule to obtain the first target table structure; performs deletion processing on the fields in the initial table structure according to the field deletion rule to obtain the second target table structure; performs addition processing on the fields in the initial table structure according to the field addition rule to obtain the third target table structure; and determines the target data structure according to the first target table structure, the second target table structure, and the third target table structure.

[0086] To elaborate, Figure 5 yes Figure 3 The flowchart of another exemplary embodiment of step S330 in the data processing method shown is as follows, wherein when the preset data conversion rules include field renaming rules, field modification rules, field addition rules and field deletion rules, step S330 includes the following steps:

[0087] Step S510: rename the fields in the initial table structure according to the field renaming rule to obtain a first target table structure.

[0088] The detailed steps of step S510 may refer to step S410 and will not be repeated here.

[0089] Step S520: Change the initial values ​​in the initial array according to the field modification rule to obtain the target values.

[0090] The detailed steps of step S520 may refer to step S420 and will not be repeated here.

[0091] Step S530: Delete the fields in the first table structure according to the field deletion rule to obtain a second target table structure.

[0092] The detailed steps of step S530 may refer to step S430 , wherein the first table structure in step S530 is processed as the initial table structure of step S430 , and will not be repeated here.

[0093] Step S540: Add fields in the second table structure according to the field adding rule to obtain a third target table structure.

[0094] The detailed steps of step S540 may refer to step S440 , wherein the second table structure in step S540 is processed as the initial table structure of step S440 , and will not be repeated here.

[0095] Step S550: determine the target data structure according to the third target table structure and the target value.

[0096] Specifically, the data processing module stores the target values ​​in the initial array, stores the initial values ​​in the initial array as other target values ​​in the initial array, and obtains the target array; the storage position of the third target table structure in the data processing module and the target array are determined as the target data structure.

[0097] In one embodiment, the data processing module uses a preset data transformation rule in the operator to perform rule transformation on the initial table structure. Figure 6 As shown in the first figure, the initial table structure includes A int id=0; B varchar id=1; C int id=2; among them, in A int id=0, A is the field name, int indicates the field type corresponding to A, and id=0 indicates the field offset corresponding to A; in B varchar id=1, B is the field name, varchar indicates the field type corresponding to B, and id=1 indicates the field offset corresponding to B; in C int id=2, C is the field name, int indicates the field type corresponding to C, and id=2 indicates the field offset corresponding to C. The initial table structure is passed to the add field operator, and the field addition rules in the add field operator include adding a D field, a field type of int, and a field offset of 3; after the add field operator processes the initial table structure according to the field addition rules, the first candidate table structure is obtained, as shown in FIG. Figure 6As shown in the second figure in , the first candidate table structure includes A int id=0; Bvarchar id=1; C int id=2; D int id=3; and D int id=3 is newly added. The first target table structure is passed to the field renaming operator. The field renaming rule in the field renaming operator includes renaming field A to E. Then the field renaming operator processes the first candidate table structure to obtain the second candidate table structure, as shown in Figure 6 As shown in the third figure in , the second candidate table structure includes E int id=0; B varchar id=1; C int id=2; D int id=3; the field name A in the second candidate table structure is changed to E. The second candidate table structure is passed to the field deletion operator. The field deletion rule in the field deletion operator includes deleting the C field. The field deletion operator processes the second candidate table structure to obtain the third candidate table structure, as shown in Figure 6 As shown in the fourth figure in the third candidate table structure, the third candidate table structure includes E int id=0; B varchar id=1; D int id=3; the C field is deleted in the third candidate table structure. The third candidate table structure is passed to the field calculation operator. The field modification rule in the field calculation operator includes D field+1. The field calculation operator processes the third candidate table structure to obtain the target table structure, as shown in Figure 6 As shown in the fifth figure, the target table structure includes E int id=0; B varchar id=1; D int id=3; the target table structure does not change, and the initial value of the field offset 3 at the position in the initial array is increased by 1 to obtain the target value.

[0098] The step of the data processing module retrieving the target value from the target data structure and writing it into the second data storage module according to the data storage rule of the second data storage module includes: obtaining the data to be written according to the target value retrieved from the target data structure; in response to the target storage format of the second data storage module being inconsistent with the format of the data to be written, performing format conversion on the data to be written according to the target storage format to obtain the converted data; and writing the converted data into the second data storage module.

[0099] The data to be written is data waiting to be written into the second data storage module.

[0100] The data processing module obtains the data to be written according to the target value retrieved from the target data structure. The target data structure includes the storage position of the target table structure in the data processing module and the target array, and the target array includes target values ​​with different values. Specifically, the data processing module determines the target storage field and the field offset corresponding to the target storage field from the target table structure according to the storage position of the target table structure in the data processing module; retrieves the target value from the target array according to the field offset; and performs format conversion processing on the target value according to the target storage field to obtain the data to be written. It should be noted that the data processing module stores the initial value stored in the initial array as the target value in the target array.

[0101] The target storage field refers to a field to be stored in the target table structure. The target storage field may include a field name, and one or more of a type, a length, a precision, and a time format.

[0102] In one embodiment, the data processing module finds the target table structure according to the storage position of the target table structure in the data processing module, and determines the target storage field in the target table structure and the field offset corresponding to the target storage field; retrieves the target value at the storage position represented by the field offset from the target array; and converts the target value into a format according to the format in the target storage field to obtain the data to be written.

[0103] Furthermore, in response to the target storage format of the second data storage module being consistent with the format of the data to be written, the data processing module writes the data to be written into the second data storage module.

[0104] It can be seen that by converting the format of the data to be written, it is possible to avoid the second data storage module being incompatible with the data to be written, thereby achieving storage of the data to be written in the second data storage module.

[0105] Figure 7 FIG. 1 is a block diagram of a data processing device shown in an exemplary embodiment of the present application. Figure 7 As shown, the exemplary data processing device 700 includes: an acquisition module 710, an initial array determination module 720, a target data structure construction module 730 and a writing module 740. The data processing device 700 is applied to a data processing system, and the data processing system includes a data processing module, a first data storage module and a second data storage module. One end of the data processing module is connected to the first data storage module, and the other end is connected to the second data storage module. The data processing module exists in the data processing device. Specifically:

[0106] The acquisition module 710 is used to acquire the data to be stored from the first data storage module according to the acquired data storage instruction, wherein the data to be stored includes the initial value to be stored and the field to be stored corresponding to the initial value.

[0107] The initial array determination module 720 is used to store the initial values ​​into a preset array to obtain an initial array, wherein the initial array includes initial values ​​with different values.

[0108] The target data structure construction module 730 is used to construct a target data structure according to the initial array and the fields to be stored corresponding to each initial value.

[0109] The writing module 740 is used to retrieve the target value from the target data structure and write it into the second data storage module according to the data storage rule of the second data storage module.

[0110] In this exemplary data processing device, the data to be stored is obtained from the first data storage module according to the acquired data storage instruction, and the data to be stored includes the initial value to be stored and the field to be stored corresponding to the initial value; the initial value is stored in a preset array to obtain an initial array, and the initial array includes initial values ​​with different values; a target data structure is constructed according to the initial array and the fields to be stored corresponding to each initial value; and the target value is retrieved from the target data structure and written into the second data storage module according to the data storage rule of the second data storage module. On the one hand, the target value is called from the target data structure constructed by the initial array and the field to be stored, and then written into the second data storage module, so as to realize the transfer of the stored data. On the other hand, the use of the initial array to store initial values ​​with different values ​​can avoid the repeated storage of the same value, reduce the storage space of the data, and thus realize the reduction of the storage space of the data in the process of transferring the stored data.

[0111] Among them, the functions of each module can be found in the data processing method embodiment, and will not be repeated here.

[0112] In order to implement the data processing method of the above embodiment, the present application proposes another electronic device, for details, please refer to Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0113] The electronic device 800 includes a memory 801 and a processor 802 , wherein the memory 801 and the processor 802 are coupled.

[0114] The memory 801 is used to store program data, and the processor 802 is used to execute the program data to implement the data processing method of the above embodiment.

[0115] In this embodiment, the processor 802 may also be referred to as a CPU (Central Processing Unit). The processor 802 may be an integrated circuit chip having signal processing capabilities. The processor 802 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor 802 may also be any conventional processor, etc.

[0116] The present application also provides a computer-readable storage medium, such as Fig. 9 As shown, the computer-readable storage medium 900 is used to store program data 901. When the program data 901 is executed by the processor, it is used to implement the data processing method in the method embodiment of the present application.

[0117] The method involved in the data processing method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0118] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing system, the data processing system includes a data processing module, a first data storage module and a second data storage module, one end of the data processing module is connected to the first data storage module, and the other end is connected to the second data storage module, the method includes: The data processing module acquires the data to be stored from the first data storage module according to the acquired data storage instruction, wherein the data to be stored includes an initial value to be stored and a field to be stored corresponding to the initial value; Storing the initial values ​​in a preset array to obtain an initial array, wherein the initial array includes initial values ​​with different values; Constructing a target data structure according to the initial array and the fields to be stored corresponding to each initial value; According to the data storage rule of the second data storage module, a target value is retrieved from the target data structure and written into the second data storage module; The step of constructing a target data structure according to the initial array and the fields to be stored corresponding to each initial value comprises: Determine a field offset according to a storage position of the initial value in the initial array; Constructing an initial table structure corresponding to the field to be stored according to the field to be stored corresponding to the initial value and the field offset corresponding to the initial value; Perform rule-changing processing on the initial table structure and / or initial array according to preset data transformation rules to obtain a target data structure; The preset data transformation rule includes at least one of a field renaming rule, a field modification rule, a field adding rule and a field deleting rule.

2. The data processing method according to claim 1, characterized in that: The step of storing the initial values ​​into a preset array to obtain the initial array comprises: Determine whether there is a value in the preset array that is the same as the initial value to be stored; If it does not exist, the initial value is stored in the preset array to obtain an initial array; If it exists, the same value is used as the initial value stored in the preset array, and the storage position of the same value in the preset array is used as the storage position of the initial value in the preset array to obtain the initial array.

3. The data processing method according to claim 1, characterized in that: The step of performing rule-changing processing on the initial table structure and / or initial array according to the preset data transformation rule to obtain the target data structure includes: Rename the fields in the initial table structure according to the field renaming rule to obtain a first target table structure; and / or, Performing a change process on the initial value in the initial array according to the field modification rule to obtain a target value; and / or, Deleting the fields in the initial table structure according to the field deletion rule to obtain a second target table structure; and / or, Performing a new process on the fields in the initial table structure according to the field adding rule to obtain a third target table structure; A target data structure is determined according to the first target table structure, the second target table structure, the third target table structure and / or the target value.

4. The data processing method according to claim 3, characterized in that: The field renaming rule includes a first field name and a target field name corresponding to the first field name, and the step of performing renaming processing on the fields in the initial table structure according to the field renaming rule to obtain a first target table structure includes: Matching the field name to be replaced from the initial table structure according to the first field name; The to-be-replaced field names in the initial table structure are replaced with corresponding target field names to obtain a first target table structure.

5. The data processing method according to claim 3, characterized in that: The field modification rule includes a second field name and an algorithm corresponding to the second field name, and the step of performing a change process on the initial value in the initial array according to the field modification rule to obtain a target value includes: Matching a field name to be processed from the initial table structure according to the second field name; Determine the initial value to be modified from the initial array according to the field offset corresponding to the field name to be processed in the initial table structure; The initial value to be modified is calculated according to the algorithm corresponding to the second field name to obtain the target value.

6. The data processing method according to claim 1, characterized in that: The step of retrieving the target value from the target data structure and writing it into the second data storage module according to the data storage rule of the second data storage module comprises: Acquire the data to be written according to the target value retrieved from the target data structure; In response to the target storage format of the second data storage module being inconsistent with the format of the data to be written, converting the format of the data to be written according to the target storage format to obtain converted data; The converted data is written into the second data storage module.

7. The data processing method according to claim 6, characterized in that: The target data structure includes a storage location of a target table structure in the data processing module and a target array, wherein the target array includes target values ​​with different values, and the step of obtaining the data to be written according to the target value retrieved from the target data structure includes: Determining a target storage field and a field offset corresponding to the target storage field from the target table structure according to a storage position of the target table structure in the data processing module; Retrieving the target value from the target array according to the field offset; The target value is format converted according to the target storage field to obtain the data to be written.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 7.

Images