Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of big data, and discloses a data processing method and device, computer equipment, a storage medium and a program product. The method is applied to a NiFi cluster, and comprises the following steps: acquiring first target data; wherein the first target data is used for indicating changed data; performing type conversion processing on the first target data to obtain second target data; performing field mapping on the second target data to obtain third target data; and writing the third target data into an open-source data lake storage framework in a java client mode.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, computer equipment, storage medium and program product. Background Art

[0002] Data ingestion can be the process of importing data from various data sources into a big data storage system. The data ingestion process usually relies on Flink and Spark clusters.

[0003] Currently, submitting and running tasks on a Flink or Spark cluster usually involves the following steps: Developers need to write data processing task code that complies with the Flink or Spark API specifications. Package the written task code into an executable jar file or other format. Configure the Flink or Spark cluster, including parameters such as resource allocation and task scheduling. Submit the task to the cluster for execution through command line tools or APIs. During the task running process, you need to observe the running log on the cluster to monitor the execution of the task and promptly identify potential problems.

[0004] However, the process of task submission and operation involves multiple steps and configurations, which increases the complexity of the system. This requires developers to have a high level of technical skills and rich experience to ensure the correct submission and smooth operation of tasks. Summary of the invention

[0005] In view of this, the present invention provides a data processing method, apparatus, computer equipment, storage medium and program product.

[0006] In a first aspect, the present invention provides a data processing method, which is applied to a NiFi cluster, and the method includes: obtaining first target data; wherein the first target data is used to indicate the data that has changed; performing type conversion processing on the first target data to obtain second target data; performing field mapping on the second target data to obtain third target data; and writing the third target data into an open source data lake storage framework through a java client.

[0007] The data processing method provided in this embodiment collects and processes the changed data through the NiFi cluster, without requiring developers to directly write complex Flink or Spark task codes, and thus without requiring developers to have high technical levels and rich experience. At the same time, the efficiency of writing data into the open source data lake storage framework can be improved through the distributed capabilities of the NiFi cluster and the Java client method.

[0008] In a possible implementation, before acquiring the first target data, the method further includes: acquiring a source data set; when a change event is triggered, determining the changed source data from the source data set; and using the changed source data as the first target data.

[0009] The data processing method provided in this embodiment can capture the changed source data in real time when the source data changes, thereby ensuring the accuracy and timeliness of the source data. At the same time, the change event trigger mechanism allows the system to process only when the data changes, avoiding unnecessary full data processing, thereby improving processing efficiency.

[0010] In a possible implementation, field mapping is performed on the second target data to obtain the third target data, including: identifying all source fields in the source data and first attribute information corresponding to each source field; defining the target field corresponding to the open source data lake storage framework and the second attribute information corresponding to the target field according to preset target requirements; comparing the source field and the target field one by one to determine the mapping relationship; and converting each field in the second target data one by one according to the mapping relationship to obtain the third target data.

[0011] The data processing method provided in this embodiment can accurately determine the mapping relationship between source fields and target fields by comparing them one by one, thereby ensuring the accuracy of data conversion while avoiding data mismatch or loss.

[0012] In a possible implementation, the third target data is written into an open source data lake storage framework through a Java client, including: writing the third target data into an intermediate message queue; detecting whether there is data to be transmitted in the intermediate message queue; if there is no data to be transmitted in the intermediate message queue, writing the third target data into the open source data lake storage framework through a Java client.

[0013] The data processing method provided in this embodiment uses an intermediate message queue to realize asynchronous processing of data, that is, no real-time synchronization is required between data producers (such as data source systems) and consumers (such as data lake storage frameworks), thereby improving the response speed and throughput of the system.

[0014] In addition, the message queue can be used as a data buffer to balance the load between data producers and consumers, avoiding performance problems caused by consumers processing large amounts of data.

[0015] In a possible implementation, the method also includes: if there is data to be transmitted in the intermediate message queue, determining the priority of the data to be transmitted and the third target data respectively; if the priority of the third target data is higher than the priority of the data to be transmitted, writing the third target data into an open source data lake storage framework through a Java client.

[0016] The data processing method provided in this embodiment can allocate resources more reasonably through priority judgment, avoiding wasting resources on unnecessary data. When high-priority data arrives, the system can respond and process it quickly, thereby improving the resource utilization efficiency of the overall system.

[0017] In a second aspect, the present invention provides a data processing device, including a first acquisition module for acquiring first target data; wherein the first target data is used to indicate data that has changed; a conversion processing module for performing type conversion processing on the first target data to obtain second target data; a field mapping module for performing field mapping on the second target data to obtain third target data; and a writing module for writing the third target data into an open source data lake storage framework through a java client.

[0018] In one possible implementation, the second acquisition module is used to acquire a source data set; the first determination module is used to determine the changed source data from the source data set when a change event is triggered; and the second determination module is used to use the changed source data as the first target data.

[0019] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.

[0021] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of a data processing method according to an embodiment of the present invention;

[0025] Figure 3 is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0028] Data ingestion can be the process of importing data from various data sources into a big data storage system. The data ingestion process usually relies on Flink and Spark clusters.

[0029] Currently, submitting and running tasks on a Flink or Spark cluster usually involves the following steps: Developers need to write data processing task code that complies with the Flink or Spark API specifications. Package the written task code into an executable jar file or other format. Configure the Flink or Spark cluster, including parameters such as resource allocation and task scheduling. Submit the task to the cluster for execution through command line tools or APIs. During the task running process, you need to observe the running log on the cluster to monitor the execution of the task and promptly identify potential problems.

[0030] However, the process of task submission and operation involves multiple steps and configurations, which increases the complexity of the system. This requires developers to have a high level of technical skills and rich experience to ensure the correct submission and smooth operation of tasks.

[0031] Based on this, the present invention provides a data processing method, which collects and processes the changed data through the NiFi cluster, without the need for developers to directly write complex Flink or Spark task codes, thereby eliminating the need for developers to have high technical levels and rich experience. At the same time, the efficiency of writing data into the open source data lake storage framework can be improved through the distributed capabilities of the NiFi cluster and the Java client method.

[0032] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] In this embodiment, a data processing method is provided, which can be used in a NiFi cluster. Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0034] Step S101, obtaining first target data; wherein the first target data is used to indicate data that has changed.

[0035] The first target data may be used to indicate the data that has changed. The first target data may exist in a specific format (such as JSON, XML, CSV, etc.) and include information about the data change. Specifically, the first target data may be obtained from a data source such as a database, a log file, a data storage queue, etc.

[0036] As an example, the CDC component of NiFi is used to read the first target data from a database, and the first target data may also be read from a log file.

[0037] Step S102, performing type conversion processing on the first target data to obtain second target data.

[0038] The second target data may represent data obtained after type conversion processing, and its format or type meets the requirements of subsequent processing steps. The type conversion processing may indicate the process of converting the first target data from one data format or type to another data format or type.

[0039] As an example, for data stored in text form (such as JSON, XML, etc.), you can use the corresponding parsing library to convert it into a format that is easy to process (such as a Java object). Convert the data into a specific format (such as date format, number format, etc.) as required. Convert string-type numbers to integer or floating-point types. For example: The first target data is a string in JSON format, which contains a field called "age" whose value is a string-type number (such as "25"). In order to convert it to a format more suitable for subsequent processing, you can write a type conversion logic to convert the value of the field from a string type to an integer type. The converted second target data will be a Java object or other format of data containing an integer type "age" field.

[0040] As an example, in the type conversion process, the first target data can be regarded as a raw, low-level data representation, while the second target data is an abstracted and encapsulated high-level data representation. For example, the first target data may be a series of raw sensor readings, while the second target data may be physical quantities such as temperature and humidity that have been calculated and converted, and these physical quantities are encapsulated in a higher-level data structure, such as an object or structure containing multiple attributes. Data conversion tasks can be defined and executed using data conversion tools (such as Apache Camel, Talend, etc.); programming languages ​​(such as Java, Python, etc.) can also be used to write custom data conversion scripts or functions. In addition, some databases and storage systems also provide built-in data conversion functions, such as CAST and CONVERT functions in SQL.

[0041] Step S103, performing field mapping on the second target data to obtain third target data.

[0042] The third target data may refer to data obtained after field mapping, whose fields and formats fully meet the requirements of the data lake storage framework. Field mapping may be the process of matching and converting the fields in the second target data with the expected fields in the data lake storage framework. Specifically, it may include operations such as changing the field name, converting the field type, and processing the field value.

[0043] As an example, write field mapping logic to match the fields in the second target data with the fields in the data lake storage framework; use data conversion tools such as Apache Camel to perform field conversion. For example: convert the order_date field in the Java object to the orderDate field in the data lake storage framework, and convert the date format from yyyy-MM-dd to yyyyMMdd.

[0044] Step S104: write the third target data into the open source data lake storage framework through the Java client.

[0045] Java Client may refer to a client program written in the Java programming language for interacting with an open source data lake storage framework. It provides APIs and methods for writing data to the data lake storage framework. An open source data lake storage framework may refer to an open source system for storing large-scale data, such as Hadoop HDFS, Apache Parquet, Amazon S3, etc. It provides high availability, scalability, and persistence of data.

[0046] As an example, use Hadoop's FileSystem API to write data to HDFS; use Amazon S3's Java SDK to write data to S3; and use Apache Parquet's Java API to write data to Parquet files.

[0047] The data processing method provided in this embodiment collects and processes the changed data through the NiFi cluster, without requiring developers to directly write complex Flink or Spark task codes, and thus without requiring developers to have high technical levels and rich experience. At the same time, the efficiency of writing data into the open source data lake storage framework can be improved through the distributed capabilities of the NiFi cluster and the Java client method.

[0048] In a possible implementation, before step S101, the method further includes:

[0049] Step a1: Obtain source data set.

[0050] The source data set may represent a collection of raw data and may be stored in a database, a file system, or other data storage system. Specifically, SQL statements may be used to extract data from a database. File read APIs (such as Java's Files class and Python's open function) may also be used to read data from a file. Data may also be obtained from an external API through an HTTP request, etc., which are not specifically limited here and may be implemented by those skilled in the art.

[0051] Step a2: when a change event is triggered, the changed source data is determined from the source data set.

[0052] A change event may be an event that causes the data in the source data set to change. These events may be triggered by a variety of factors, such as user operations, system updates, changes in external data sources, etc. Specifically, a trigger may be set in the database to automatically perform a specific operation (such as recording a change log) when the data changes. The system log or application log may be analyzed to find out the operation that caused the data change. The current data may be compared with the last acquired data periodically or in real time to find out the differences, etc., which are not specifically limited here and may be implemented by those skilled in the art.

[0053] Step a3: taking the changed source data as the first target data.

[0054] After the changed source data is determined, the changed source data is used as the first target data.

[0055] The data processing method provided in this embodiment can capture the changed source data in real time when the source data changes, thereby ensuring the accuracy and timeliness of the source data. At the same time, the change event trigger mechanism allows the system to process only when the data changes, avoiding unnecessary full data processing, thereby improving processing efficiency.

[0056] In a possible implementation, the above step S103 includes:

[0057] Step b1, identifying all source fields in the source data and the first attribute information corresponding to each source field.

[0058] The source field can represent the data items contained in each data record in the source data set, such as user name, age, address, etc. The first attribute information can represent information describing the characteristics of the source field, such as data type (string, integer, floating point number, etc.), data length, whether it is allowed to be empty, etc. Specifically, use database management tools or APIs to query table structures. Use NiFi clusters to parse file formats, etc.

[0059] Step b2: define the target field corresponding to the open source data lake storage framework and the second attribute information corresponding to the target field according to the preset target requirements.

[0060] The preset target requirements can be used to characterize the requirements of matching the open source data lake storage framework, that is, the data after mapping needs to meet the requirements of writing to the open source data lake storage framework. The target field can be used to characterize the various fields used to store the converted data in the open source data lake storage framework. The second attribute information is used to describe the characteristics of the target field. Specifically, you can refer to the documentation of the data lake storage framework to define fields and attributes, and you can also use the API or tools provided by the data lake storage framework to define fields. For example: define the target fields "user_name" (string type), "user_age" (integer type), and "user_address" (string type) in Hadoop HDFS or Amazon S3.

[0061] Step b3: compare the source fields and the target fields one by one to determine the mapping relationship.

[0062] By comparing the names, data types and other attributes of the source field and the target field, the mapping relationship between them is determined. Specifically, the mapping relationship can be manually established according to the field name and type. It is also possible to use a data mapping tool (such as Apache Nifi, Talend, etc.) to automatically establish the mapping relationship, etc., which is not specifically limited here and can be implemented by those skilled in the art.

[0063] Step b4: convert each field in the second target data one by one according to the mapping relationship to obtain the third target data.

[0064] According to the mapping relationship, each field in the second target data (i.e., the data after the changed source data is initially processed) is converted to meet the requirements of the data lake storage framework. Specifically, data conversion tools (such as Apache Camel, Spring Integration, etc.) can be used to convert fields, and cdc components in the NiFi cluster can also be used for conversion, etc., which are not specifically limited here and can be implemented by those skilled in the art.

[0065] The data processing method provided in this embodiment can accurately determine the mapping relationship between source fields and target fields by comparing them one by one, thereby ensuring the accuracy of data conversion while avoiding data mismatch or loss.

[0066] In a possible implementation, the above step S104 includes:

[0067] Step c1, writing the third target data into the intermediate message queue.

[0068] The intermediate message queue can be a queue system for asynchronously transmitting data between different systems or components, which is used to decouple data producers and consumers and improve the scalability and fault tolerance of the system. Specifically, the converted third target data (i.e., data that meets the requirements of the data lake storage framework) is written into the intermediate message queue for subsequent data transmission and processing.

[0069] As an example, you can use the synchronous or asynchronous send methods of the Kafka Producer API. You can also use the publish method of the RabbitMQ Java client. You can also use the Java clients of other message queues such as ActiveMQ and RocketMQ.

[0070] Step c2, detecting whether there is data to be transmitted in the intermediate message queue.

[0071] Regularly check or monitor the intermediate message queue to determine whether there is new third target data written into the intermediate message queue. Once written into the intermediate message queue, the third target data needs to be processed. Before the third target data is processed, it is necessary to detect whether there is data to be transmitted in the intermediate message queue, that is, whether there is data in the intermediate message queue that needs to be transmitted but has not been transmitted yet.

[0072] Step c3: If there is no data to be transmitted in the intermediate message queue, the third target data is written into the open source data lake storage framework through the Java client.

[0073] If there is no data to be transmitted in the intermediate message queue, indicating that the third target data can be directly written into the open source data lake storage framework, the third target data can be written into the open source data lake storage framework through the Java client.

[0074] The data processing method provided in this embodiment uses an intermediate message queue to realize asynchronous processing of data, that is, no real-time synchronization is required between data producers (such as data source systems) and consumers (such as data lake storage frameworks), thereby improving the response speed and throughput of the system.

[0075] In addition, the message queue can be used as a data buffer to balance the load between data producers and consumers, avoiding performance problems caused by consumers processing large amounts of data.

[0076] In a possible implementation, the method further includes:

[0077] Step d1: If there is data to be transmitted in the intermediate message queue, the priorities of the data to be transmitted and the third target data are determined respectively.

[0078] Check whether there is data to be transmitted in the intermediate message queue. If so, determine the priority of the data and the third target data respectively.

[0079] As an example, the API provided by the message queue is used to query the number or content of messages in the queue to determine the priority, and the priority of the data is determined according to the business logic or preset rules. For example, the priority can be set according to the urgency, importance or size of the data, etc., which is not specifically limited here and can be implemented by those skilled in the art.

[0080] For example, if Kafka is used as a message queue, the number of messages in the queue can be queried through Kafka's Consumer API. Then, the priority of each message can be determined based on the timestamp or business logic of the message.

[0081] Step d2: If the priority of the third target data is higher than the priority of the data to be transmitted, the third target data is written into the open source data lake storage framework through the Java client.

[0082] The priority of the third target data is compared with the priority of the data to be transmitted. If the priority of the third target data is higher, the third target data is written into the open source data lake storage framework using the Java client.

[0083] The data processing method provided in this embodiment can allocate resources more reasonably through priority judgment, avoiding wasting resources on unnecessary data. When high-priority data arrives, the system can respond and process it quickly, thereby improving the resource utilization efficiency of the overall system.

[0084] Please refer to Figure 2 , Figure 2 It is a schematic diagram of a data processing method provided according to an embodiment of the present invention.

[0085] Figure 2 Oracle is the database for the data processing method. There are multiple nodes in the NiFi cluster. During the distributed task processing, each node can perform the corresponding task. For the data collection stage, the corresponding data can be collected from the database. Then the data type of the changed data is converted through the node of the next process, and the field mapping of the data after the data type conversion is performed through the node of the next process, and then the data after the field mapping is written into hudi (open source data lake storage framework).

[0086] The data processing method provided in this embodiment collects and processes the changed data through the NiFi cluster, without requiring developers to directly write complex Flink or Spark task codes, and thus without requiring developers to have high technical levels and rich experience. At the same time, the efficiency of writing data into the open source data lake storage framework can be improved through the distributed capabilities of the NiFi cluster and the Java client method.

[0087] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0088] This embodiment provides a data processing device, such as Figure 3 As shown, it includes: a first acquisition module 301, used to acquire first target data; wherein the first target data is used to indicate the data that has changed; a conversion processing module 302, used to perform type conversion processing on the first target data to obtain second target data; a field mapping module 303, used to perform field mapping on the second target data to obtain third target data; a writing module 304, used to write the third target data into the open source data lake storage framework through a java client.

[0089] In one possible implementation, the device also includes: a second acquisition module, used to acquire a source data set; a first determination module, used to determine the changed source data from the source data set when a change event is triggered; and a second determination module, used to use the changed source data as the first target data.

[0090] In a possible implementation, the field mapping module 303 includes: an identification unit, which is used to identify all source fields in the source data and the first attribute information corresponding to each source field; a definition unit, which is used to define the target field corresponding to the open source data lake storage framework and the second attribute information corresponding to the target field according to the preset target requirements; a first determination unit, which is used to compare the source field and the target field one by one to determine the mapping relationship; and a conversion unit, which is used to convert each field in the second target data one by one according to the mapping relationship to obtain the third target data.

[0091] In one possible implementation, the writing module 304 includes: a first writing unit, used to write the third target data into the intermediate message queue; a first detection unit, used to detect whether there is data to be transmitted in the intermediate message queue; and a second writing unit, used to write the third target data into the open source data lake storage framework through the Java client if there is no data to be transmitted in the intermediate message queue.

[0092] In a possible implementation, the device further includes: a priority determination module, which is used to determine the priority of the data to be transmitted and the third target data respectively if there is data to be transmitted in the intermediate message queue; and a priority writing module, which is used to write the third target data into an open source data lake storage framework through a Java client if the priority of the third target data is higher than the priority of the data to be transmitted.

[0093] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0094] The data processing device in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0095] The embodiment of the present invention also provides a computer device having the above Figure 3 The data processing device shown.

[0096] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0097] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0098] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0099] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0101] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0102] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a source downloaded through a network and stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0103] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0104] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that: The method is applied to a NiFi cluster; wherein the method comprises: Acquire first target data; wherein the first target data is used to indicate the data that has changed; Performing type conversion processing on the first target data to obtain second target data; Performing field mapping on the second target data to obtain third target data; The third target data is written into the open source data lake storage framework through the Java client.

2. The data processing method according to claim 1, characterized in that: Before acquiring the first target data, the method further includes: Get the source dataset; When a change event is triggered, determining the changed source data from the source data set; The changed source data is used as the first target data.

3. The data processing method according to claim 2, characterized in that: Performing field mapping on the second target data to obtain third target data includes: Identify all source fields in the source data and first attribute information corresponding to each source field; According to the preset target requirements, define the target fields corresponding to the open source data lake storage framework and the second attribute information corresponding to the target fields; Compare the source field and the target field one by one to determine a mapping relationship; According to the mapping relationship, each field in the second target data is converted one by one to obtain third target data.

4. The data processing method according to claim 1, characterized in that: Writing the third target data into the open source data lake storage framework through the Java client includes: Writing the third target data into the intermediate message queue; Detecting whether there is data to be transmitted in the intermediate message queue; If there is no data to be transmitted in the intermediate message queue, the third target data is written into the open source data lake storage framework through the Java client.

5. The data processing method according to claim 4, characterized in that: The method further comprises: If there is data to be transmitted in the intermediate message queue, respectively determine the priority of the data to be transmitted and the third target data; If the priority of the third target data is higher than the priority of the data to be transmitted, the third target data is written into the open source data lake storage framework through the Java client.

6. A data processing device, characterized in that: The device comprises: A first acquisition module, used to acquire first target data; wherein the first target data is used to indicate the data that has changed; A conversion processing module, used for performing type conversion processing on the first target data to obtain second target data; A field mapping module, used for performing field mapping on the second target data to obtain third target data; The writing module is used to write the third target data into the open source data lake storage framework through the Java client.

7. The data processing device according to claim 6, characterized in that: The device also includes: A second acquisition module is used to acquire a source data set; A first determining module, configured to determine the changed source data from the source data set when a change event is triggered; The second determining module is used to use the changed source data as the first target data.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the data processing method according to any one of claims 1 to 5.

Images