A data processing method and device
By identifying the message format and calling the configuration file to generate adapted Flink data flow, the problems of poor data processing compatibility and low efficiency in the prior art are solved, and efficient data preprocessing and system performance improvement are achieved.
Patent Information
- Application Number
- CN202010771119.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-03
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-08-03
AI Technical Summary
The lack of tools for automatically identifying and processing data formats in the prior art leads to poor compatibility and inefficiency in data processing, and developers need to manually operate to convert data into a format that Flink can process.
By obtaining messages, identifying message formats, calling predefined configuration files to obtain field information, generating DataStream or Table data streams, and converting data streams based on configuration files to adapt to data formats that Flink can process.
Improve development efficiency, reduce costs, enhance data preprocessing quality, ensure system performance, and adapt to the processing needs of multiple data sources.
Smart Images

Figure CN113779018B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art
[0002] In the field of big data development technology, there are many real-time computing frameworks, such as Storm, Spark, and Flink (Flink is an open source processing engine for batch data and stream data). When receiving and processing messages, it is often necessary to parse the content, process abnormal data, and then map it into a structured table after various conversions. Faced with a lot of parsed content and field information, it is necessary to map them one by one.
[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:
[0004] Currently, there is no known tool that can automatically identify the format of the data to be processed and process the data, converting it into a data format that Flink can subsequently operate on, such as DataStream or Table. The usual approach is for actual developers to perform corresponding operations based on the known value format, process dirty data, and convert the data into a format that Flink can correctly process. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a data processing method and device, which can solve the problems of poor compatibility and low efficiency of existing data processing methods.
[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising obtaining a message and identifying a message format; according to the message format, calling a predefined configuration file to obtain field information corresponding to the message to generate a corresponding data stream; based on a return data structure preset in the configuration file, converting and returning the data stream.
[0007] Optionally, get messages including:
[0008] Use FlinkKafkaConsumer to connect to the corresponding Kafka topic and get messages.
[0009] Optionally, identify the message format, including:
[0010] Parse the message through the fastjson method to determine whether the parsing is successful. If so, the message format is json format, otherwise the message format is String format or other format.
[0011] Optionally, calling a predefined configuration file to obtain field information corresponding to the message includes:
[0012] Recursively parse the message according to the configuration file corresponding to the message format to obtain field information.
[0013] Optionally, the message is parsed recursively according to a configuration file corresponding to the message format to obtain field information, including:
[0014] If the message format is JSON, the message is parsed recursively using the fastjson method to obtain the corresponding key-value pairs according to the configuration file; or
[0015] If the message format is string format, the corresponding field information of the message is obtained according to the configuration file; wherein, if the field information is in json format, the fastjson method is used to recursively parse the field information to obtain the corresponding key-value pairs.
[0016] Optionally, a corresponding data stream is generated, including:
[0017] The addScource method is called to generate a data stream in the DataStream data format according to the field information.
[0018] Optionally, based on a return data structure preset in the configuration file, converting and returning the data stream includes:
[0019] Obtaining a return data structure preset in the configuration file;
[0020] Determine whether the returned data structure is of table type. If so, use the toTable method to convert the data stream into a table type data stream and return it. Otherwise, directly return the generated data stream.
[0021] In addition, the present invention also provides a data processing device, including an acquisition module, used to acquire messages and identify message formats; a processing module, used to call a predefined configuration file according to the message format to obtain field information corresponding to the message to generate a corresponding data stream; based on the return data structure preset in the configuration file, the data stream is converted and returned.
[0022] An embodiment of the above invention has the following advantages or beneficial effects: the field information given in the custom configuration file is matched with the Kafka stream data and the field information, and then converted into a DataStream stream to adapt to this data format, and a DataStream stream that meets business requirements is generated, and finally the data stream is returned in combination with the format required to be returned in the user configuration file. Thus, the present invention improves the work efficiency of developers, reduces development costs and subsequent maintenance costs, and increases output. In addition, the quality of data preprocessing is improved, dirty data is properly processed, and system performance is guaranteed. In addition, various data sources can be properly processed to ensure that subsequent data processing by Flink proceeds normally.
[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0025] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;
[0026] Figure 2 is a schematic diagram of data flow according to an embodiment of the present invention;
[0027] Figure 3 is a schematic diagram of the main flow of middleware data processing according to an embodiment of the present invention;
[0028] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;
[0029] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0030] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0032] Figure 1is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method includes:
[0033] Step S101, obtaining a message and identifying the message format.
[0034] In some embodiments, FlinkKafkaConsumer is used to connect to the corresponding Kafka topic to obtain messages. Preferably, FlinkKafkaConsumer010 is used to connect to the corresponding Kafka topic to obtain messages. Kafka is an open source stream processing platform.
[0035] Preferably, when identifying the message format, the message can be parsed by the fastjson method to determine whether the parsing is successful. If so, the message format is json format, otherwise the message format is string format or other formats. Among them, fastjson is a json framework that can convert strings that conform to the json specification into json format.
[0036] It is worth noting that if Figure 2 As shown, according to the identified message format, the corresponding consumer program can be connected to avoid multiple judgments of the message format and affect performance. In addition, the consumer program can be connected to the flink data source preprocessing tool middleware to transfer the input message to the flink data source preprocessing tool middleware for preprocessing (i.e., step S102 and step S103), and return the processed message to the consumer program. Among them, flink is a stream computing engine. Stream generally refers to streaming media technology.
[0037] Step S102: According to the message format, a predefined configuration file is called to obtain field information corresponding to the message to generate a corresponding data stream.
[0038] In some embodiments, the message is parsed recursively according to the configuration file corresponding to the message format to obtain the field information. The configuration file can be stored in a resource folder, and different message formats correspond to different pre-defined configuration files.
[0039] In a further embodiment, if the message format is json format, the fastjson method is used to fully parse the message in a recursive manner (the field can be parsed to find out that the value in the key-value pair format corresponding to the key) to obtain the corresponding key-value pair according to the configuration file. The key-value pair can be generally referred to as a Map collection, which is commonly used in the actual development process, such as click streams, order messages, etc., and is a data structure that conforms to {"key", "value"}.
[0040] If the message format is string format, the corresponding field information of the message is obtained according to the configuration file. For example, according to the string separator given in the configuration file and the segments of the string where the required field is located and the name information of each segment, the corresponding field information of the message is obtained. Separators include "," "|" "&" and the like. If the field information is in json format, the fastjson method is used to parse the field information recursively to obtain the corresponding key-value pairs.
[0041] As some other embodiments, the addScource method is called to generate a data stream in DataStream data format according to the field information, wherein DataStream is a data sequence called by the instruction stream, including input data and intermediate results.
[0042] Step S103: based on the return data structure preset in the configuration file, convert the data stream and return it.
[0043] In some embodiments, when executing step S103, the return data structure preset in the configuration file can be obtained. Then, it is determined whether the return data structure is of table type. If so, the toTable method is used to convert the data stream into a table type data stream and return it for subsequent data operations. Otherwise, the generated data stream is directly returned for subsequent data operations. Among them, toTable can be used when using flinkSQL to process real-time data to convert DataStream into a table.
[0044] In summary, the present invention creatively proposes a method for processing source data self-adaptive structuring based on flink, that is, converting the data transmitted by message middleware such as kafka into data structure formats such as DataStream or table that can be processed by flink later, which can adapt to various business scenario requirements and has extremely high compatibility and scalability. In addition, dirty data in the source data can be processed efficiently, realizing a more accurate data preprocessing function and significantly improving system performance.
[0045] Figure 3 1 is a schematic diagram of the main flow of middleware data processing according to an embodiment of the present invention. The data processing of the middleware (such as the flink data source preprocessing tool middleware) includes:
[0046] Receive the incoming data (i.e., message), obtain the message format of the incoming data in the judgment string format component, and then call the corresponding configuration file according to the message format (i.e., obtain the user's configuration information). Based on the configuration file, obtain the field information corresponding to the message. If the message format is in json format, recursively parse the message using the fastjson method to obtain the corresponding key-value pairs according to the configuration file. Or if the message format is in string format, obtain the corresponding field information of the message according to the configuration file (i.e., obtain the field information and combine it). Wherein, if the field information is in json format, recursively parse the field information using the fastjson method to obtain the corresponding key-value pairs.
[0047] According to the combined field information or key-value pairs, the addScource method is called to generate a data stream in DataStream data format. Then the return data structure preset in the configuration file is obtained to determine whether the return data structure is of table type. If so, the toTable method is used to convert the data stream into a table type data stream and return it. Otherwise, the generated data stream is directly returned.
[0048] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention, such as Figure 4 As shown, the data processing device 400 includes an acquisition module 401 and a processing module 402. The acquisition module 401 acquires a message and identifies the message format; the processing module 402 calls a predefined configuration file according to the message format to obtain field information corresponding to the message to generate a corresponding data stream; based on the return data structure preset in the configuration file, the data stream is converted and returned
[0049] In some embodiments, the acquisition module 401 acquires a message, including:
[0050] Use FlinkKafkaConsumer to connect to the corresponding Kafka topic and get messages.
[0051] In some embodiments, the acquisition module 401 identifies the message format, including:
[0052] Parse the message through the fastjson method to determine whether the parsing is successful. If so, the message format is json format, otherwise the message format is String format or other format.
[0053] In some embodiments, the processing module 402 calls a predefined configuration file to obtain field information corresponding to the message, including:
[0054] Recursively parse the message according to the configuration file corresponding to the message format to obtain field information.
[0055] In some embodiments, the processing module 402 parses the message recursively according to the configuration file corresponding to the message format to obtain field information, including:
[0056] If the message format is JSON, the message is parsed recursively using the fastjson method to obtain the corresponding key-value pairs according to the configuration file; or
[0057] If the message format is string format, the corresponding field information of the message is obtained according to the configuration file; wherein, if the field information is in json format, the fastjson method is used to recursively parse the field information to obtain the corresponding key-value pairs.
[0058] In some embodiments, the processing module 402 generates a corresponding data stream, including:
[0059] The addScource method is called to generate a data stream in the DataStream data format according to the field information.
[0060] In some embodiments, the processing module 402 converts and returns the data stream based on the return data structure preset in the configuration file, including:
[0061] Obtaining a return data structure preset in the configuration file;
[0062] Determine whether the returned data structure is of table type. If so, use the toTable method to convert the data stream into a table type data stream and return it. Otherwise, directly return the generated data stream.
[0063] It should be noted that the data processing method and the data processing device described in the present invention have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.
[0064] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing device, data processing determination or data processing determination device according to the embodiment of the present invention can be applied.
[0065] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0066] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
[0067] The terminal devices 501, 502, 503 may be various electronic devices having a data processing screen or a data processing determination screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0068] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.
[0069] It should be noted that the data processing method or data processing determination method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the computing device is generally disposed in the server 505 .
[0070] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0071] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0072] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0073] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal data processor (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0074] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
[0075] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0076] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0077] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, it may be described as follows: a processor includes an acquisition module and a processing module. The names of these modules do not, in some cases, limit the modules themselves.
[0078] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes obtaining a message and identifying a message format; according to the message format, calling a predefined configuration file to obtain field information corresponding to the message to generate a corresponding data stream; based on the return data structure preset in the configuration file, converting and returning the data stream.
[0079] The technical solution according to the embodiment of the present invention can solve the problems of poor compatibility and low efficiency of existing data processing methods.
[0080] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A data processing method, characterized in that: include: Get the message and identify the message format; According to the message format, a predefined configuration file is called to obtain field information corresponding to the message to generate a corresponding data stream; Based on the return data structure preset in the configuration file, convert the data stream and return it; Call the predefined configuration file to obtain the field information corresponding to the message, including: If the message format is JSON, the message is parsed recursively using the fastjson method to obtain the corresponding key-value pairs according to the configuration file; or If the message format is string format, the corresponding field information of the message is obtained according to the delimiter and required fields in the configuration file; wherein, if the field information is in json format, the fastjson method is used to recursively parse the field information to obtain the corresponding key-value pairs.
2. The method according to claim 1, characterized in that Get messages, including: Use FlinkKafkaConsumer to connect to the corresponding Kafka topic and get messages.
3. The method according to claim 2, characterized in that Identify the message format, including: Parse the message through the fastjson method to determine whether the parsing is successful. If so, the message format is json format, otherwise the message format is String format or other format.
4. The method according to claim 1, characterized in that: Generate the corresponding data flow, including: The addScource method is called to generate a data stream in the DataStream data format according to the field information.
5. The method according to any one of claims 1 to 4, characterized in that: Based on the return data structure preset in the configuration file, converting and returning the data stream includes: Obtaining a return data structure preset in the configuration file; Determine whether the returned data structure is of table type. If so, use the toTable method to convert the data stream into a table type data stream and return it. Otherwise, directly return the generated data stream.
6. A data processing device, characterized in that: include: The acquisition module is used to obtain messages and identify message formats; A processing module, configured to call a predefined configuration file to obtain field information corresponding to the message according to the message format, so as to generate a corresponding data stream; based on a return data structure preset in the configuration file, convert and return the data stream; The processing module is also used to: If the message format is JSON, the message is parsed recursively using the fastjson method to obtain the corresponding key-value pairs according to the configuration file; or If the message format is string format, the corresponding field information of the message is obtained according to the delimiter and required fields in the configuration file; wherein, if the field information is in json format, the fastjson method is used to recursively parse the field information to obtain the corresponding key-value pairs.
7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.
8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
