Data migration method and device, electronic equipment and storage medium

Abstract

The embodiment discloses a data migration method, device, equipment, system and computer storage medium, and the method comprises the following steps: receiving a character-separated value (CSV) file read by other processing units of the Nifi cluster; mapping text content of the CSV file into a Record; preprocessing the Record to obtain a preprocessed Record, and storing the preprocessed Record in a database.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data migration method, apparatus, electronic device and storage medium. Background Technology

[0002] In related technologies, data migration of Comma-Separated Values ​​(CSV) files in a NiFi cluster requires seven processing units (processors), involving file reading, filtering, text-to-database field mapping, preprocessing, etc. Each processing unit performs its function independently in the NiFi cluster. During the migration of CSV files, each processing unit consumes a large amount of disk read / write and network transmission resources, which reduces the data migration efficiency of CSV files. Summary of the Invention

[0003] This application provides a data migration method that can reduce disk read / write and network resource consumption during CSV file data migration, thereby improving data migration efficiency.

[0004] This application provides a data migration method, the method comprising:

[0005] Receive CSV files read by other processing units of the NiFi cluster;

[0006] Map the text content of the CSV file to Record records;

[0007] The Record is preprocessed to obtain a preprocessed Record, which is then stored in the database.

[0008] In some embodiments, preprocessing the Record to obtain a preprocessed Record includes:

[0009] Identify the variables obtained by concatenating fields in the Record;

[0010] The identified variables are processed according to the expression language syntax of the Nifi cluster to obtain the preprocessed Record.

[0011] In some embodiments, identifying the variables obtained by concatenating fields in the Record includes:

[0012] A fixed prefix is ​​added to the variable to identify it.

[0013] In some embodiments, processing the identified variables according to the expression language syntax of the NiFi cluster to obtain the preprocessed Record includes:

[0014] According to the expression language syntax, the identified variables are extracted and data is transformed to obtain the preprocessed Record.

[0015] In some embodiments, the step of extracting variables and transforming data from the identified variables according to the expression language syntax to obtain the preprocessed Record includes:

[0016] In the onScheduled method, the identified variable is recognized as a variable using the expression language syntax;

[0017] When the onTrigger method is called, the identified variables are extracted and transformed using the expression language syntax to obtain the preprocessed Record.

[0018] In some embodiments, mapping the text content of the CSV file to Record records includes:

[0019] The NiFi cluster's record reader is used to map the text content of the CSV file to Record records.

[0020] In some embodiments, the step of using the record reader of the NiFi cluster to map the text content of the CSV file to a record includes:

[0021] The text content of the CSV file is mapped according to the Avro format fields defined in the record reader to obtain the Record.

[0022] This application also provides a data migration apparatus, which is applied in a service processing unit in a NiFi cluster. The apparatus includes: a receiving module, a first processing module, and a second processing module, wherein...

[0023] The receiving module is used to receive CSV files read by other processing units of the NiFi cluster;

[0024] The first processing module is used to map the text content of the CSV file to Record records;

[0025] The second processing module is used to preprocess the Record to obtain the preprocessed Record, and then store the preprocessed Record in the database.

[0026] This application also provides an electronic device, including a processor and a memory for storing a computer program capable of running on the processor; wherein,

[0027] The processor is used to run the computer program to perform any of the above-described data migration methods.

[0028] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described data migration methods.

[0029] As can be seen, the embodiments of this application address the scenario of data migration of CSV files in a NiFi cluster. By redesigning the processing unit in the NiFi cluster, a single business processing unit can be used to implement the process of reading text content, data mapping, data preprocessing, and data storage. This achieves a shorter path in the data migration process, reduces the number of flowfie generation, reduces the resource consumption of disk read / write and network transmission, and improves the data migration efficiency of CSV files. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating data migration of CSV files within a NiFi cluster, as described in related technologies.

[0031] Figure 2A This is a schematic diagram of the processing unit used in NiFi clusters for data migration of CSV files in related technologies.

[0032] Figure 2B This is a diagram illustrating data preprocessing during data migration of CSV files in a NiFi cluster, as described in related technologies. Figure 1 ;

[0033] Figure 2C This is a schematic diagram (2) illustrating data preprocessing during data migration of CSV files in a NiFi cluster, as described in related technologies.

[0034] Figure 3 This is a flowchart illustrating the data migration method according to an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of different processing units when migrating CSV files in a NiFi cluster according to an embodiment of this application;

[0036] Figure 5 This is a schematic diagram showing the controller service of the record reader in an embodiment of this application;

[0037] Figure 6This is a flowchart illustrating the preprocessing of Records in this embodiment of the application;

[0038] Figure 7 This is a schematic diagram illustrating data preprocessing during data migration of a CSV file in an embodiment of this application;

[0039] Figure 8 This is a schematic diagram illustrating the configuration information of the service processing unit in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of the database connection information involved in the business processing unit in this application embodiment;

[0041] Figure 10 This is a comparative diagram showing the effects of data migration on CSV in the embodiments of this application and related technologies;

[0042] Figure 11A This diagram illustrates the relationship between the amount of data processed and the time in related technologies using NiFi clusters. Figure 1 ;

[0043] Figure 11B This is a schematic diagram illustrating the relationship between the amount of data processed by a NiFi cluster and time in related technologies (Figure 2).

[0044] Figure 12 This is a schematic diagram of the data migration apparatus according to an embodiment of this application;

[0045] Figure 13 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0046] In related technologies, NiFi is an easy-to-use, powerful, and reliable data processing and distribution system. Designed for data streams, NiFi supports highly configurable data routing, transformation, and system mediation logic for indicator graphs, and allows dynamic data retrieval from multiple data sources. NiFi's design goal is to automate data flow between systems; within a NiFi cluster, data migration into a database can be achieved using CSV files. Here, CSV files store tabular data (numbers and text) in plain text format. The text content of a CSV file is a sequence of characters and does not contain data that must be interpreted like binary numbers. A CSV file consists of any number of records, separated by some kind of newline character; each record consists of fields.

[0047] Reference Figure 1 In related technologies, the process of migrating CSV files in a NiFi cluster may include:

[0048] Step 101: Monitor the directory.

[0049] In practical applications, refer to Figure 2A The NiFi cluster's processing unit 1 can monitor the directory. Processing unit 1 can monitor the directory based on the Secure File Transfer Protocol (SFTP), which provides a secure network encryption method for transferring files.

[0050] Step 102: Read the file.

[0051] Reference Figure 2A The processing unit 2 in the Nifi cluster can read files from the file list in the directory. For example, the processing unit 2 can read files based on SFTP.

[0052] Step 103: File filtering.

[0053] Reference Figure 2A Processing unit 3 can filter files in the current directory based on filenames, thereby achieving file filtering.

[0054] Step 104: Read the text content.

[0055] Here, processing unit 3 can also read the text content of the filtered file after the file is filtered.

[0056] Step 105: Data mapping.

[0057] In practical applications, refer to Figure 2A Processing unit 4 can parse the text content of the CSV file into Record records, and extract the columns that need to be mapped from the parsed Record records according to the column matching information in the configuration file. After extracting the columns that need to be mapped, processing unit 4 can perform column mapping on the extracted columns to obtain new Record records.

[0058] Step 106: Data preprocessing.

[0059] In practical applications, refer to Figure 2A Processing unit 5 performs field concatenation on the new Record record generated by processing unit 4 using a path matching (Record Path Value) strategy, obtaining the concatenated field result. This concatenated result is then used as a new Record record for the next processing unit to access. (See reference...) Figure 2B Processing unit 5 can read the new Record generated by processing unit 4 and store the read data as a fixed field value. Then, it can perform field concatenation on the fixed field value and store the variable obtained by field concatenation.

[0060] Reference Figure 2Aand Figure 2C Processing unit 6 can perform field calculations using a variable matching (Literal Value) strategy on the field concatenation result (i.e., the variable obtained by concatenating fields) to obtain the field calculation result; the field calculation result can be used as a new Record for the next processing unit to call. For example, the field calculation strategy includes numerical calculations and logical judgments on the field.

[0061] It can be seen that processing unit 5 and processing unit 6 are different processing units that perform data preprocessing operations. Processing unit 5 and processing unit 6 will use different record update strategies to implement the preprocessing of Record records. That is, the record update strategy adopted by processing unit 5 is the path matching strategy, and the record update strategy adopted by processing unit 6 is the variable matching strategy.

[0062] Step 107: Map into the database.

[0063] In practical applications, processing unit 7 can match the new Record generated by processing unit 6 with the format of the table in the database, and then insert the Record into the database by inserting it row by row.

[0064] In related technologies, during the data migration of CSV files within a NiFi cluster, the high availability mechanism implemented by the NiFi cluster itself causes a flowfile to be generated for each new business processing unit (e.g., a Record record). This results in data being written to disk multiple times, consuming significant amounts of disk I / O and network transmission resources. Furthermore, while the controller service of the processing unit can perform data preprocessing on Record records mapped to database fields (here, the controller service is a built-in service within the processing unit), in related technologies, field concatenation and field calculation can only be implemented based on different processing units, further consuming disk resources.

[0065] It can be seen that during the migration of CSV files, each processing unit consumes a large amount of disk read / write and network transmission resources, which reduces the data migration efficiency of CSV files; the data migration efficiency of CSV files is usually lower than that of directly writing Java programs for data migration.

[0066] In summary, how to reduce the flowfile generated in the NiFi cluster, reduce disk resource consumption, and further reduce the difficulty of mapping and processing CSV files to database fields are technical problems that urgently need to be solved.

[0067] In view of the above-mentioned technical problems, the technical solutions of the embodiments of this application are proposed.

[0068] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the embodiments of the present application can be implemented in any combination.

[0069] This application provides a data migration method applicable to a service processing unit in a NiFi cluster, which can be referred to as an InsertDatabaseRecord Processor. This service processing unit can be implemented using electronic devices, which may include terminals and / or servers. Here, the terminal can be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a programmable consumer electronics product, a network PC, a minicomputer system, etc. The server can be a minicomputer system, a mainframe computer system, or a distributed cloud computing environment including any of the above systems, etc.

[0070] Servers and other electronic devices may include program modules that execute computer instructions. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0071] Figure 3 This is a flowchart illustrating the data migration method according to an embodiment of this application. Figure 1 ,like Figure 1 As shown, the process may include:

[0072] Step 301: Receive the CSV file read by other processing units in the NiFi cluster.

[0073] In this embodiment, other processing units may include a processing unit for monitoring the directory, and a processing unit for reading files and filtering files; see reference Figure 4 Processing unit A is a processing unit used to monitor the directory. Processing unit A has the same function as processing unit 1, and will not be described again here. Processing unit B is a processing unit used to read files and filter files. Processing unit B can pass the filtered files to the business processing unit.

[0074] Step 302: Map the text content of the CSV file to Record records.

[0075] In this embodiment of the application, the business processing unit can read the text content of the filtered file, and then, referring to step 105, perform data mapping on the read text content to obtain a Record.

[0076] Step 303: Preprocess the Record to obtain the preprocessed Record, and store the preprocessed Record in the database.

[0077] In this embodiment of the application, the business processing unit can sequentially perform field concatenation and field calculation on the Record to obtain the preprocessed Record.

[0078] In practical applications, steps 301 to 303 can be implemented based on a processor in an electronic device. This processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, or microprocessor. It is understood that other electronic devices can also implement the above processor functions, and this application embodiment does not impose any limitations.

[0079] As can be seen, the embodiments of this application address the scenario of data migration of CSV files in a NiFi cluster. By redesigning the processing unit in the NiFi cluster, a single business processing unit can be used to implement the process of reading text content, data mapping, data preprocessing, and data storage. This achieves a shorter path in the data migration process, reduces the number of flowfie generation, reduces the resource consumption of disk read / write and network transmission, and improves the data migration efficiency of CSV files.

[0080] In some embodiments, the Nifi cluster's own RecordReader can be used in the business processing unit to map the text content of a CSV file to a Record. Figure 5 A detailed schematic diagram of the controller service for the record reader.

[0081] As can be seen, a single business processing unit can be used to implement the process of reading text content and mapping data, thereby reducing the number of times flowfie is generated, reducing the resource consumption of disk read / write and network transmission, and improving the data migration efficiency of CSV files.

[0082] In some embodiments, the business processing unit can map the text content of a CSV file to obtain a Record based on the Avro format fields defined in the record reader.

[0083] Here, Avro is a programming language-independent serialization format. Avro data is defined using a language-independent schema, which is described using JSON. The data is serialized into a binary file or a JSON file, typically using a binary file. Avro requires the schema when reading and writing files, and the schema is usually embedded within the data file.

[0084] In practical applications, the data type of the text content in a CSV file can be identified based on the Avro format fields. Here, the data type of the text content in the CSV file can be integer, string, etc. After identifying the data type of the text content in the CSV file, the text content in the CSV file can be mapped to Record records based on the data type of the text content. That is, the column mapping of the text content in the CSV file can be performed based on the Avro format fields to obtain Record records.

[0085] For example, embodiments of this application can convert the text content of a CSV file into a Record record based on the Avro format field defined in the Schema text (Text) of the record reader for subsequent operations, which is easy to implement.

[0086] In some embodiments, refer to Figure 6 The process of preprocessing Records can include:

[0087] Step 3031: Identify the variables obtained by concatenating fields in the Record record.

[0088] In this embodiment of the application, the business processing unit can use a path matching strategy to concatenate fields; for example, a fixed prefix can be added to the variable obtained by concatenating fields to identify the variable; it can be seen that the identification of the variable can be achieved conveniently and quickly by adding a fixed prefix.

[0089] For example, a variable can be identified by adding a fixed prefix "_var" before it. It should be noted that the fixed prefix in this embodiment can also be any prefix other than "_var", and this is not limited thereto.

[0090] Step 3032: Process the identified variables according to the Expression Language syntax of the Nifi cluster to obtain the preprocessed Record.

[0091] Here, the primary function of the NiFi cluster is data processing and distribution. In the NiFi cluster, data is abstracted as flowfiles, which consist of two parts: content and attributes. The content is the actual data to be processed and distributed; the flowfile doesn't directly store the data but rather stores a pointer to the actual storage address. Attributes are metadata and descriptive information related to the data itself, in the form of key-value pairs. Flowfile attributes allow us to perform many data processing operations without needing to read the data content. NiFi introduces an expression language to enable users to easily access flowfile attributes and perform operations such as referencing, comparing, and assigning values ​​to them.

[0092] For example, the identified variables can be extracted and transformed according to the expression language syntax to obtain the preprocessed Record.

[0093] In practical applications, identified variables can be recognized as column variables in expression language syntax, thereby enabling variable extraction and data transformation of identified variables through expression language syntax.

[0094] In some embodiments, the identified variable can be recognized as a variable using the expression language syntax in the onScheduled method; here, the onScheduled method represents a method for executing a scheduled task.

[0095] When the onTrigger method is called, the identified variables are extracted and transformed using the expression language syntax to obtain the preprocessed Record record; here, the onTrigger method represents the method to execute the trigger task.

[0096] As can be seen, the embodiments of this application can process the identified variables by variable identification and according to the expression language syntax. Since the variables obtained by field concatenation do not need to be used as new Record records for the next processing unit to call, the process of field concatenation and field calculation can be implemented in the same processing unit. This avoids the problem in related technologies that require two processing units to implement field concatenation and field calculation, reduces data reading and writing between different processing units, and reduces disk and network resource consumption caused by generating multiple flowfiles.

[0097] Furthermore, the business processing unit combines the advantages of processing unit 5 and processing unit 6 mentioned above. It identifies variables obtained by concatenating fields using a fixed prefix, and then processes the variables obtained by concatenating fields according to... Figure 2C The expression language syntax shown is used for calculation and processing to obtain a preprocessed Record; for example, the preprocessed Record can be processed according to... Figure 7 The database column names on the left are mapped to enable the preprocessed Record records to be stored in the database. This avoids the redundant flowfile caused by using multiple processing units with the same technology.

[0098] In this embodiment, a single business processing unit can implement the processes of reading text content, data mapping, data preprocessing, and data storage. Figure 8 This is a schematic diagram illustrating the configuration information of the service processing unit in an embodiment of this application. (Refer to...) Figure 8 The configuration information for the business processing unit includes settings, plans, attributes, and explanations. Figure 9 This is a schematic diagram of the database connection information involved in the business processing unit in this application embodiment, referring to... Figure 9 Database connection information includes database-related driver information, database name, username and password, connection timeout, and other relevant information.

[0099] The effects of the data migration method according to the embodiments of this application will be illustrated below. In the first example, the NiFi cluster is a 3-node cluster (each node has 16 cores, 64GB of memory, and 500GB of disk space), and the records to be processed are 100 million CSV records. Without any preprocessing operations during the data migration process, the data migration method of related technologies is used to migrate the CSV files, referring to... Figure 10 In the first example, the data migration from the CSV file to the database took 90 minutes.

[0100] In the second example, the NiFi cluster is a 3-node cluster using PostgreSQL master-slave mode. Each node has 8 cores, 32GB of memory, and 500GB of disk space. It needs to process 100 million CSV records. Without any preprocessing during the data migration process, the average migration speed for the CSV files using relevant data migration methods is 10GB / 5min. With preprocessing performed during the data migration process, the migration speed for the CSV files using relevant data migration methods is as follows: [Refer to...] Figure 10 In the second example, the time taken to migrate the data from the CSV file to the database was 180 minutes, with an average migration speed of 5GB / 5 minutes. Figure 11A and Figure 11B In the diagram, the horizontal axis represents time, and the vertical axis represents the amount of data processed. Curve 1 represents the total amount of data processed by all nodes in the Nifi cluster, while curves 2 to 4 represent the amount of data processed by the first, second, and third nodes of the Nifi cluster, respectively.

[0101] In the third example, the NiFi cluster is a 3-node cluster (each node has 16 cores, 64GB of memory, and 500GB of disk space), and it needs to process 100 million CSV records. The data migration method of this application embodiment is used to migrate the CSV files, referring to... Figure 10 In the second example, the time to migrate the CSV file data to the database was 35 minutes, the average data migration speed was 26GB / 5 minutes, and the stable migration speed was 38GB / 5 minutes. In summary, through... Figure 10 , Figure 11A and Figure 11B As can be seen, compared with the data migration methods of related technologies, the data migration method of this application embodiment improves the processing speed by about 5 times.

[0102] Based on the data migration method proposed in the foregoing embodiments, this application also proposes a data migration apparatus; this apparatus is applied in the service processing unit of a Nifi cluster. Figure 12 This is a schematic diagram of the composition structure of a data migration device according to an embodiment of this application, as shown below. Figure 12 As shown, the device may include a receiving module 120, a first processing module 121, and a second processing module 122, wherein,

[0103] The receiving module 120 is used to receive CSV files read by other processing units of the NiFi cluster;

[0104] The first processing module 121 is used to map the text content of the CSV file to Record records;

[0105] The second processing module 122 is used to preprocess the Record to obtain a preprocessed Record, and store the preprocessed Record in the database.

[0106] In some embodiments of this application, the second processing module 122 is used to preprocess the Record to obtain a preprocessed Record, including:

[0107] Identify the variables obtained by concatenating fields in the Record;

[0108] The identified variables are processed according to the expression language syntax of the Nifi cluster to obtain the preprocessed Record.

[0109] In some embodiments of this application, the second processing module 122 is used to identify variables obtained by concatenating fields in the Record, including:

[0110] A fixed prefix is ​​added to the variable to identify it.

[0111] In some embodiments of this application, the second processing module 122 is used to process the identified variables according to the expression language syntax of the NiFi cluster to obtain the preprocessed Record, including:

[0112] According to the expression language syntax, the identified variables are extracted and data is transformed to obtain the preprocessed Record.

[0113] In some embodiments of this application, the second processing module 122 is configured to perform variable extraction and data transformation on the identified variables according to the expression language syntax to obtain the preprocessed Record, including:

[0114] In the onScheduled method, the identified variable is recognized as a variable using the expression language syntax;

[0115] When the onTrigger method is called, the identified variables are extracted and transformed using the expression language syntax to obtain the preprocessed Record.

[0116] In some embodiments of this application, the first processing module 121 is configured to map the text content of the CSV file to Record records, including:

[0117] The NiFi cluster's record reader is used to map the text content of the CSV file to Record records.

[0118] In some embodiments of this application, the first processing module 121 is used to map the text content of the CSV file to a Record using the record reader of the NiFi cluster, including:

[0119] The text content of the CSV file is mapped according to the Avro format fields defined in the record reader to obtain the Record.

[0120] In practical applications, the receiving module 120, the first processing module 121, and the second processing module 122 can all be implemented using a processor of an electronic device. The processor can be at least one of an ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, or microprocessor. It is understood that other electronic devices can also implement the above-mentioned processor functions, and this application embodiment does not impose any limitations.

[0121] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0122] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0123] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the data migration methods provided in this application embodiment.

[0124] Accordingly, this application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the data migration methods provided in the above embodiments.

[0125] This application also provides an electronic device. Figure 13 A schematic diagram of an optional component structure of the electronic device provided in an embodiment of this application is shown below. Figure 13 As shown, the electronic device 130 includes:

[0126] Memory 131 is used to store executable instructions;

[0127] The processor 132 is used to implement any of the above-described data migration methods when executing executable instructions stored in the memory 131.

[0128] The processor 132 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0129] The aforementioned computer-readable storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0130] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0131] It should be understood that the phrase "some embodiments" mentioned throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0135] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0136] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the device automatic test line to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0137] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0138] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0139] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data migration method, characterized in that, The method, applied to a service processing unit in a NiFi cluster, includes: Receive character-separated value CSV files read by other processing units of the NiFi cluster; Based on the Avro format fields defined in the record reader of the NiFi cluster, the text content of the CSV file is mapped to Record records; A fixed prefix is ​​added to the variables obtained by concatenating fields in the Record to identify the variables; The identified variables are processed according to the expression language syntax of the Nifi cluster to obtain preprocessed Record records, which are then stored in the database.

2. The data migration method according to claim 1, characterized in that, The process of processing the identified variables according to the expression language syntax of the NiFi cluster to obtain preprocessed Record records includes: According to the expression language syntax, the identified variables are extracted and data is transformed to obtain the preprocessed Record.

3. The data migration method according to claim 2, characterized in that, The step of extracting variables and transforming data from the identified variables according to the expression language syntax to obtain the preprocessed Record includes: In the onScheduled method, the identified variable is recognized as a variable using the expression language syntax; When the onTrigger method is called, the identified variables are extracted and transformed using the expression language syntax to obtain the preprocessed Record.

4. A data migration device, characterized in that, The device is used in the service processing unit of a NiFi cluster. The device includes: a receiving module, a first processing module, and a second processing module, wherein... The receiving module is used to receive character-separated value CSV files read by other processing units of the NiFi cluster; The first processing module is used to map the text content of the CSV file into Record records according to the Avro format fields defined in the record reader of the Nifi cluster; The second processing module is used to add a fixed prefix to the variables obtained by concatenating fields in the Record to identify the variables; The identified variables are processed according to the expression language syntax of the Nifi cluster to obtain preprocessed Record records, which are then stored in the database.

5. An electronic device, characterized in that, Includes a processor and memory for storing computer programs that can run on the processor; wherein, The processor is used to run the computer program to perform the data migration method according to any one of claims 1 to 3.

6. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the data migration method according to any one of claims 1 to 3.

Images