Data format conversion method, device, system, electronic device and storage medium
By customizing data format conversion algorithms and page operations, the difficulties in interface integration caused by inconsistent data formats between different systems were resolved, achieving efficient data format conversion and improving development efficiency.
Patent Information
- Application Number
- CN202110531564.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-05-14
AI Technical Summary
Data transmission between different systems is hampered by inconsistent interfaces and data formats, requiring extensive manual development of interfaces for integration, resulting in a large workload and low efficiency.
This paper provides a data format conversion method. By obtaining the data format and customizing the format conversion algorithm, the conversion process and algorithm can be selected through page display, thereby achieving flexible conversion between data formats and avoiding duplicate coding.
It enables free conversion between different formats, reduces the workload of developers, improves development efficiency, and simplifies the conversion process through intuitive page operations.
Smart Images

Figure CN113159970B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the financial field, and more specifically, to a data format conversion method, device, system, electronic device and storage medium. Background Art
[0002] An enterprise involves many systems, and data forwarding is often required between systems. Since the data transmission interfaces between different systems are not the same, data transmission between systems often encounters such a scenario: the collection and reporting system A collects data from other business systems B, C, and D in real time, and then the collection and reporting system A forwards the collected data to each data receiving system E, F, and G. Since the collection interfaces and collection data formats of different collection and reporting systems A vary greatly, and the interface formats of business systems B, C, D and data receiving systems E, F, and G are also different, manual development is often required to connect the interface of the collection and reporting system A with the interface of systems B, C, D, E, F, and G, which is a lot of work. Summary of the invention
[0003] In view of this, the present disclosure provides a data format conversion method, device, system, electronic device and storage medium.
[0004] One aspect of the present disclosure provides a data format conversion method, including: acquiring first data; acquiring a first data format of the first data and a second data format to be converted into the first data; customizing a format conversion algorithm between the first data format and the second data format; and converting the first data into second data having a second data format according to the format conversion algorithm.
[0005] According to an embodiment of the present disclosure, the customized format conversion algorithm between the first data format and the second data format includes: formulating a conversion process according to the conversion rules for converting the first data format into the second data format; according to each step of the conversion process, selecting the data processing algorithm required to execute each step from a preset data processing algorithm library to form the format conversion algorithm.
[0006] According to an embodiment of the present disclosure, the steps of the conversion process are selected in sequence through the step drop-down box displayed on the page to constitute the conversion process; the data processing algorithm corresponding to each step is selected through the algorithm drop-down box displayed on the page to constitute the format conversion algorithm.
[0007] According to an embodiment of the present disclosure, the steps in the conversion process include at least: splitting the first data into multiple data columns; intercepting a character string from each of the data columns; selecting a keyword from the character string and converting the keyword into a keyword that satisfies the second data format; reconnecting and combining the converted keyword with the corresponding data in the data column to generate the second data.
[0008] According to an embodiment of the present disclosure, the steps in the conversion process also include: filtering or reverse filtering the data column to screen a specified string or regular expression book; and / or splicing multiple data columns; and / or adding prefixes or suffixes to the data column or the string; and / or removing redundant strings.
[0009] According to an embodiment of the present disclosure, obtaining the first data includes: obtaining the first data from an upstream data system; storing the first data in a Kafka queue; taking out the first data from the Kafka queue in sequence and storing them in a first database for data format conversion.
[0010] According to an embodiment of the present disclosure, the acquiring of the first data from the upstream data system includes: when the upstream data system detects that there is a data change in a second database contained therein, receiving the changed data in the second database as the first data.
[0011] According to an embodiment of the present disclosure, the method further includes: storing the second data in a third database so that the second data can be searched.
[0012] According to an embodiment of the present disclosure, the method further includes: forwarding the second data to a downstream data system.
[0013] On the other hand, the present disclosure provides a data format conversion device, including: a data acquisition module, used to acquire first data; a format acquisition module, used to acquire a first data format of the first data and a second data format to be converted into the first data; a conversion algorithm customization module, used to customize a format conversion algorithm between the first data format and the second data format; a data conversion module, used to convert the first data into second data having a second data format according to the format conversion algorithm.
[0014] According to an embodiment of the present disclosure, the conversion algorithm customization module includes: a conversion rule customization unit, which is used to formulate a conversion process according to the conversion rules for converting the first data format into the second data format; a conversion algorithm combination unit, which is used to select the data processing algorithms required to execute each step from a preset data processing algorithm library according to each step of the conversion process, to form the format conversion algorithm.
[0015] According to an embodiment of the present disclosure, the conversion algorithm customization module also includes: a process customization page, which is used to display a step drop-down box, allowing the user to select each step of the conversion process in sequence to constitute the conversion process; an algorithm selection page, which is used to display an algorithm drop-down box, allowing the user to select the data processing algorithm corresponding to each step to constitute the format conversion algorithm.
[0016] Another aspect of the present disclosure provides a computer system, comprising: a data format conversion device as described in any one of the second aspects.
[0017] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods described in the first aspect.
[0018] Another aspect of the present disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement any one of the methods of the first aspect.
[0019] According to the embodiments of the present disclosure, first, a conversion process is customized according to the format of the data itself and the format into which the data is to be converted, and then a corresponding conversion algorithm is generated according to the conversion process, so that the data is converted into a specified format according to the customized conversion process. This method can realize the customization of the conversion process, making the conversion method between data formats flexible and changeable, thereby realizing free conversion between different formats, avoiding the need for developers to perform different encodings for different data interfaces, and eliminating the need for developers to repeatedly write interfaces, greatly reducing the workload of developers and improving development efficiency. In addition, the implementation of this method can be achieved through page display, and the optional process steps and data processing algorithms for executing each step displayed on the page make the application of this method simple and easy to operate, and the display of the conversion process and algorithm is more intuitive and highly operable. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0021] Figure 1 The application scenario of the data format conversion method according to the embodiment of the present disclosure is schematically shown;
[0022] Figure 2 An exemplary system architecture for the data format conversion method and apparatus disclosed herein is schematically shown;
[0023] Figure 3A flowchart of a data format conversion method according to an embodiment of the present disclosure is schematically shown;
[0024] Figure 4A The flowchart of step S330 of the data format conversion method according to the embodiment of the present disclosure is schematically shown;
[0025] Figure 4B The implementation diagram of step S330 according to the embodiment of the present disclosure is schematically shown;
[0026] Figure 5 A schematic diagram schematically shows a conversion process according to an embodiment of the present disclosure;
[0027] Figure 6 A schematic diagram schematically shows a conversion process according to another embodiment of the present disclosure;
[0028] Figure 7 A flowchart of a data format conversion method according to another embodiment of the present disclosure is schematically shown;
[0029] Fig. 8A A block diagram of a data format conversion device according to an embodiment of the present disclosure is schematically shown;
[0030] Figure 8B A block diagram schematically shows a conversion algorithm customization module of a data format conversion device according to an embodiment of the present disclosure;
[0031] Figure 8C A block diagram schematically shows a conversion algorithm customization module of a data format conversion device according to another embodiment of the present disclosure;
[0032] Fig. 9 A schematic diagram schematically shows a computer system according to an embodiment of the present disclosure;
[0033] Fig.10 A block diagram of an electronic device 1000 according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0035] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0036] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0037] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0038] Embodiments of the present disclosure provide a data format conversion method, device, computer system, electronic device and computer-readable storage medium. The method includes two parts: customizing format conversion rules and format conversion. In customizing format conversion rules, developers can customize the conversion process of converting first data in a first data format into second data in a second data format according to their own needs, and determine the algorithm for executing each step in the conversion process. Based on the received identity recognition signal, multiple robots can recognize each other. After completing the customization of the conversion rules, enter the format conversion process to convert the first data into the second data. The method can flexibly customize the format conversion rules according to actual needs, with strong adaptability and high degree of freedom.
[0039] Figure 1 The application scenario of the data format conversion method according to the embodiment of the present disclosure is schematically shown.
[0040] like Figure 1As shown, in this application scenario, the collection and reporting system A will collect data from business systems B, C, and D in real time, where the data formats of business systems B, C, and D may all be different. The data is converted and sorted by the collection and reporting system A and sent to business systems E, F, and G, where the collection interfaces and collection data formats of business systems E, F, and G are different. Under this condition, in order to enable effective data transmission between business systems, developers need to develop structures that can communicate with each other for each interacting business system, and even need to develop data format conversion codes separately. Due to the large number of data formats, if conversion codes between different data formats are developed one by one, a lot of manpower and development time will be consumed, and the development cost is too high.
[0041] Figure 2 The exemplary system architecture 200 to which the data format conversion method and device according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 2 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0042] Figure 2 An exemplary system architecture to which the method and apparatus for data format conversion of the present disclosure can be applied is schematically shown.
[0043] like Figure 2 As shown, the system architecture 200 according to this embodiment may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired and / or wireless communication links, etc.
[0044] The user can use the terminal devices 201, 202, 203 to interact with the server 205 through the network 204 to receive or send data, etc. Various communication client applications can be installed on the terminal devices 201, 202, 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples), and the data formats of various applications can be the same or different.
[0045] The terminal devices 201 , 202 , and 203 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0046] The server 205 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using terminal devices 201, 202, and 203. The background management server may perform format conversion and other processing on received data such as user requests, and forward the processed data (such as web pages, information, or data obtained or generated according to user requests) to other terminal devices.
[0047] It should be noted that the data format conversion method provided in the embodiment of the present disclosure can generally be executed by the server 205. Accordingly, the data format conversion device provided in the embodiment of the present disclosure can generally be arranged in the server 205. The data format conversion method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 205 and can communicate with the terminal devices 201, 202, 203 and / or the server 205. Correspondingly, the data format conversion device provided in the embodiment of the present disclosure can also be arranged in a server or server cluster that is different from the server 205 and can communicate with the terminal devices 201, 202, 203 and / or the server 205. Alternatively, the data format conversion method provided in the embodiment of the present disclosure can also be executed by the terminal devices 201, 202, or 203, or can also be executed by other terminal devices different from the terminal devices 201, 202, or 203. Correspondingly, the data format conversion device provided in the embodiment of the present disclosure may also be arranged in the terminal device 201 , 202 , or 203 , or in other terminal devices different from the terminal device 201 , 202 , or 203 .
[0048] For example, the first data may be originally stored in any one of the terminal devices 201, 202, or 203 (for example, the terminal device 201, but not limited thereto), or stored on an external storage device and may be imported into the terminal device 201. Then, the terminal device 201 may locally execute the data format conversion method provided in the embodiment of the present disclosure, or send the first data to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the first data may execute the data format conversion method provided in the embodiment of the present disclosure.
[0049] It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0050] Figure 3 The following schematically shows a flow chart of a data format conversion method according to an embodiment of the present disclosure.
[0051] like Figure 3As shown, a data format conversion method provided by an embodiment of the present disclosure includes operations S310 to S340.
[0052] In operation S310, first data is acquired.
[0053] In the embodiment of the present disclosure, the first data may be any data having a row of data format, which is obtained from a business system that generates the first data.
[0054] In operation S320, a first data format of the first data and a second data format to be converted into the first data are acquired.
[0055] In the embodiments of the present disclosure, the data formats of the first data and the second data may be known to the developer, or the device or equipment that executes the data format conversion method may analyze the first data based on pre-stored and local data format types after obtaining the first data to obtain the data format type of the first data. The second data format may be appropriately determined based on the data adapted by the interface for receiving the first data.
[0056] In operation S330, a format conversion algorithm between the first data format and the second data format is customized.
[0057] In the disclosed embodiment, the format conversion algorithm is specifically customized according to operations S331 to S332. The customization process does not require writing conversion codes for conversions between different data formats one by one. Instead, the conversion steps are written separately, and the conversion process between different formats is obtained by combining multiple conversion steps to form a format conversion algorithm. Based on operation S330, different format conversion algorithms can be flexibly generated for different format conversions to achieve free conversion between any two formats. This method is highly flexible and easy to operate, and can greatly improve the development efficiency of developers.
[0058] In operation S340, the first data is converted into second data having a second data format according to the format conversion algorithm.
[0059] According to the data format conversion method provided by the embodiment of the present disclosure, first, the conversion process is customized according to the format of the data itself and the format to which the data is to be converted, and then, the corresponding conversion algorithm is generated according to the conversion process, so that the data is converted to the specified format according to the customized conversion process. This method can realize the customization of the conversion process, making the conversion method between data formats flexible and changeable, and then realizing the free conversion between different formats, avoiding the need for developers to perform different encodings for different data interfaces, so that developers do not have to repeatedly write interfaces, greatly reducing the workload of developers and improving development efficiency. In addition, the implementation of this method can be realized through page display. The optional process steps and the data processing algorithms for executing each step displayed on the page make the application of this method simple and easy to operate, and the display of the conversion process and algorithm is more intuitive and highly operable.
[0060] The following is a detailed description of each step in the data format conversion method provided in the embodiment of the present disclosure.
[0061] Figure 4A The flowchart of step S330 of the data format conversion method according to the embodiment of the present disclosure is schematically shown.
[0062] like Figure 4A As shown, operation S330, customizing a format conversion algorithm between the first data format and the second data format includes operations S410 to S420.
[0063] In operation S410, a conversion process is formulated according to a conversion rule for converting the first data format into the second data format.
[0064] In the disclosed embodiment, multiple basic conversion steps are preset, for example, splitting the first data into multiple data columns, extracting character strings in each data column, converting keywords in the data column into keywords in the second data format, adding prefixes or adding suffixes, etc. For conversion between different formats, there must be a conversion process, and by combining multiple basic conversion steps according to the conversion process, a complete operation process for converting between two formats can be obtained.
[0065] In operation S420, according to each step of the conversion process, a data processing algorithm required to execute each step is selected from a preset data processing algorithm library to form the format conversion algorithm.
[0066] In the disclosed embodiment, after customizing each step of the conversion process, it is necessary to set the data processing algorithm for executing the step according to the actual conversion requirements, and even preset some processing parameters for the data processing algorithm to implement the processing step. For example, to extract a specified length or a specified string from a regular expression, it is necessary to set the characteristic parameters of the recognition length and the string. The data processing algorithms of each step are combined together according to the conversion process to form a format conversion algorithm.
[0067] Figure 4B The implementation diagram of step S330 according to the embodiment of the present disclosure is schematically shown.
[0068] like Figure 4B As shown, operation S330 can be specifically implemented through a selection box provided by the page, including operations S411 to S421.
[0069] In operation S411, each step of the conversion process is selected in sequence through the step drop-down box displayed on the process customization page to form the conversion process.
[0070] In operation S421, the data processing algorithm corresponding to each of the steps is selected through the algorithm drop-down box displayed on the algorithm selection page to form the format conversion algorithm.
[0071] By providing this page, developers do not need to worry about how the backend code is implemented. They can complete the format conversion algorithm configuration by simply selecting in the drop-down box, thereby improving development efficiency.
[0072] Figure 5 The figure schematically shows a conversion process according to an embodiment of the present disclosure.
[0073] like Figure 5 As shown, the steps in the conversion process at least include operations S510 to S540.
[0074] In operation S510, the first data is split into a plurality of data columns.
[0075] In operation S520, character strings are extracted from each of the data columns.
[0076] In operation S530, a keyword is selected from the character string, and the keyword is converted into a keyword satisfying the second data format.
[0077] In operation S540, the converted keyword is reconnected and combined with the corresponding data in the data column to generate second data.
[0078] According to the embodiments of the present disclosure, each data format uses its own specific format, but generally includes keywords and data strings. Format conversion is performed, the keywords and data strings are identified, converted according to the keyword requirements of each data format, and then recombined with the data strings according to the format requirements, so that the first data can be converted from the first data format to the second data format.
[0079] In addition to keyword conversion, the steps in the conversion process can also include:
[0080] Filtering or unfiltering the data column to screen for a specified string or regular expression; and / or
[0081] splicing a plurality of the data columns; and / or
[0082] Adding a prefix or a suffix to the data column or the character string; and / or
[0083] Remove redundant strings.
[0084] Table 1 shows several commonly used data processing algorithms that can be used to implement the above steps.
[0085] Table 1
[0086]
[0087] Figure 6 The figure schematically shows a conversion process according to another embodiment of the present disclosure.
[0088] like Figure 6 As shown, another embodiment of the present disclosure illustrates a complete conversion process for format conversion. The first step is to obtain the first data that needs to be converted in format; the second step is to customize the conversion process of converting the first data into the second data; the third step is to use the split algorithm to split the data using the specified delimiter; the fourth step is to use the substring algorithm to intercept each disassembled message field; the fifth step is to use the select algorithm to select the specified required field as the key; the sixth step is to use the saveAs algorithm to change the name of the key to the desired key value; finally, the second data is generated based on the kev value.
[0089] Figure 7 The following schematically shows a flow chart of a data format conversion method according to another embodiment of the present disclosure.
[0090] like Figure 7 As shown, according to a data format conversion method provided by another embodiment of the present disclosure, in Figure 3 In the operation S310 shown, the first data is obtained, including operations S710 to S730.
[0091] In operation S710, the first data is acquired from an upstream data system.
[0092] Among them, when the upstream data system detects that there is a data change in the second database it contains, the changed data in the second database is received as the first data. Only the changed data in the upstream data system is passed to the downstream data system, which can avoid unnecessary data transmission and reduce various types of consumption. This part of the function is implemented using the open source components canal and logstash. Canal is mainly used to receive change data generated by the database Mysql; Logstash is mainly used to receive text information sent by the business system, and it supports transmission protocols such as http, logfile, tcp, etc.
[0093] In operation S720, the first data is stored in a Kafka queue.
[0094] In the disclosed embodiment, the first data is pre-stored in a Kafka queue, and the Kafka queue can be used as a buffer area, so that the system applying the data format conversion method can process the collected first data in sequence, thereby improving the processing performance of the system.
[0095] In operation S730, the first data is taken out from the Kafka queue in sequence and stored in a first database to be converted into a data format.
[0096] Storing the first data in the first database can ensure the traceability of the data, so that when a problem with data conversion occurs, the converted data can be compared with the original first data to facilitate problem location.
[0097] like Figure 7 As shown, according to a data format conversion method provided by another embodiment of the present disclosure, after generating the second data, the method further includes operations S750 to S760.
[0098] In operation S750, the second data is stored in a third database to make the second data searchable.
[0099] In operation S760, the second data is forwarded to a downstream data system.
[0100] After backing up the second data, the second data is forwarded to the downstream data system. When there is a problem with the data conversion, the first data can be compared with the second data to locate the problem and help developers adjust the conversion process. The converted data is forwarded to the downstream system, and the forwarding can support mainstream transmission methods such as http, rpc, and mq.
[0101] According to the data format conversion method provided by the embodiment of the present disclosure, first, a conversion process is customized according to the format of the data itself and the format into which the data is to be converted, and then, a corresponding conversion algorithm is generated according to the conversion process, so that the data is converted into a specified format according to the customized conversion process. The method can realize the customization of the conversion process, and by combining different data processing algorithms, such as filtering, splitting, merging, replacing, etc., a complex format conversion process is formed, so that the conversion method between data formats is flexible and changeable, thereby realizing free conversion between different formats, avoiding the need for developers to perform different encodings for different data interfaces, so that developers do not have to repeat the work of writing interfaces, greatly reducing the workload of developers and improving development efficiency. In addition, the implementation of the method can be realized through page display, and the optional process steps and data processing algorithms for executing each step displayed on the page make the application of the method simple and easy to operate, and the display of the conversion process and algorithm is more intuitive and highly operable.
[0102] Fig. 8A A block diagram of a data format conversion device according to an embodiment of the present disclosure is schematically shown.
[0103] like Fig. 8A As shown, a data format conversion device provided by an embodiment of the present disclosure includes: a data acquisition module 810, a format acquisition module 820, a conversion algorithm customization module 830, and a data conversion module 840.
[0104] A data acquisition module 810 is used to acquire first data;
[0105] A format acquisition module 820, configured to acquire a first data format of the first data and a second data format to be converted from the first data;
[0106] A conversion algorithm customization module 830, used to customize a format conversion algorithm between the first data format and the second data format;
[0107] The data conversion module 840 is used to convert the first data into second data having a second data format according to the format conversion algorithm.
[0108] Figure 8B The block diagram schematically shows a conversion algorithm customization module of the data format conversion device according to an embodiment of the present disclosure.
[0109] like Figure 8B As shown, the conversion algorithm customization module includes 830: a conversion rule customization unit 831 and a conversion algorithm combination unit 832.
[0110] The conversion rule customization unit 831 is used to formulate a conversion process according to the conversion rule for converting the first data format into the second data format.
[0111] The conversion algorithm combination unit 832 is used to select the data processing algorithm required to execute each step from the preset data processing algorithm library according to each step of the conversion process to form the format conversion algorithm.
[0112] Figure 8C A block diagram schematically shows a conversion algorithm customization module of a data format conversion device according to another embodiment of the present disclosure.
[0113] like Figure 8C As shown, the conversion algorithm customization module 830 also includes: a process customization page 833 and an algorithm selection page 834 .
[0114] The process customization page 833 is used to display a step drop-down box, allowing the user to select each step of the conversion process in sequence to form the conversion process.
[0115] The algorithm selection page 834 is used to display an algorithm drop-down box, allowing the user to select the data processing algorithm corresponding to each of the steps to form the format conversion algorithm.
[0116] Based on the conversion algorithm customization module 830, when converting the format of data, there is no need to write conversion codes for the conversion between different data formats one by one. Instead, the algorithm code of each conversion step is written separately and integrated into the corresponding page button. The conversion step is selected through the button on the page, and the conversion process between different formats is obtained by combining multiple conversion steps. Then, a complete format conversion algorithm is constructed in the background based on the algorithm associated with the button of each conversion step. Based on the conversion algorithm customization module 830, different format conversion algorithms can be flexibly generated for different format conversions, and free conversion between any two formats can be realized. This method is highly flexible and easy to operate, and can greatly improve the development efficiency of developers.
[0117] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.
[0118] For example, any multiple of the data acquisition module 810, the format acquisition module 820, the conversion algorithm customization module 830, and the data conversion module 840 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the data acquisition module 810, the format acquisition module 820, the conversion algorithm customization module 830, and the data conversion module 840 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the data acquisition module 810, the format acquisition module 820, the conversion algorithm customization module 830, and the data conversion module 840 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.
[0119] It should be noted that the data format conversion device part in the embodiment of the present disclosure corresponds to the data format conversion method part in the embodiment of the present disclosure. The description of the data format conversion device part specifically refers to the data format conversion part, which will not be repeated here.
[0120] Fig. 9 The figure schematically shows a computer system according to an embodiment of the present disclosure.
[0121] like Fig. 9 As shown, the computer system 900 includes the following Figures 8A to 8C The data format conversion device, the computer system can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc., or it can be a server or server cluster for providing various services, etc., and it can be a software system installed on these hardware devices.
[0122] Fig.10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0123] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0124] In RAM 1003, various programs and data required for the operation of system 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0125] According to an embodiment of the present disclosure, the system 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0126] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.
[0127] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0128] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0129] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .
[0130] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0131] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0132] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A data format conversion method, comprising: Acquire first data; Acquire a first data format of the first data and a second data format to be converted into the first data; Customizing a format conversion algorithm between the first data format and the second data format; According to the format conversion algorithm, the first data is converted into second data having a second data format, The customizing of the format conversion algorithm between the first data format and the second data format includes: Formulate a conversion process according to a conversion rule for converting the first data format into the second data format; According to each step of the conversion process, the data processing algorithm required to execute each step is selected from the preset data processing algorithm library to form the format conversion algorithm; The steps in the conversion process at least include: Splitting the first data into a plurality of data columns; Extracting character strings from each of the data columns; Selecting a keyword from the character string, and converting the keyword into a keyword satisfying the second data format; The converted keyword is reconnected and combined with the corresponding data in the data column to generate second data.
2. The method according to claim 1, wherein: Select each step of the conversion process in sequence through the step drop-down box displayed on the process customization page to form the conversion process; Select the data processing algorithm corresponding to each of the steps through the algorithm drop-down box displayed on the algorithm selection page to form the format conversion algorithm.
3. The method according to claim 1, wherein: The steps in the conversion process also include: Filtering or unfiltering the data column to screen for a specified string or regular expression; and / or splicing a plurality of the data columns; and / or Adding a prefix or a suffix to the data column or the character string; and / or Remove redundant strings.
4. The method according to claim 1, wherein obtaining the first data comprises: Acquire the first data from an upstream data system; Storing the first data in a Kafka queue; The first data are sequentially taken out from the Kafka queue and stored in a first database to be converted into a data format.
5. The method according to claim 4, wherein: The acquiring the first data from the upstream data system comprises: When the upstream data system detects that there is a data change in the second database contained therein, the changed data in the second database is received as the first data.
6. The method according to claim 1, further comprising: The second data is stored in a third database so that the second data can be searched.
7. The method according to claim 1, further comprising: The second data is forwarded to a downstream data system.
8. A data format conversion device, comprising: A data acquisition module, used for acquiring first data; a format acquisition module, used for acquiring a first data format of the first data and a second data format to be converted by the first data; A conversion algorithm customization module, used to customize a format conversion algorithm between the first data format and the second data format; a data conversion module, configured to convert the first data into second data having a second data format according to the format conversion algorithm, Wherein, the conversion algorithm customization module includes: A conversion rule customization unit, configured to formulate a conversion process according to a conversion rule for converting the first data format into the second data format; A conversion algorithm combination unit, used to select the data processing algorithm required to execute each step from the preset data processing algorithm library according to each step of the conversion process, to form the format conversion algorithm; The steps in the conversion process at least include: Splitting the first data into a plurality of data columns; Extracting character strings from each of the data columns; Selecting a keyword from the character string, and converting the keyword into a keyword satisfying the second data format; The converted keyword is reconnected and combined with the corresponding data in the data column to generate second data.
9. The device according to claim 8, wherein: The conversion algorithm customization module also includes: A process customization page, for displaying a step drop-down box, allowing the user to select each step of the conversion process in sequence to form the conversion process; The algorithm selection page is used to display the algorithm drop-down box, allowing the user to select the data processing algorithm corresponding to each of the steps to form the format conversion algorithm.
10. A computer system comprising: A data format conversion device as claimed in claim 8 or 9.
11. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.
