A data processing method, device, apparatus, and storage medium

Abstract

The application discloses a data processing method, device and equipment and a storage medium. The method comprises the following steps: acquiring log data; processing the log data by using an SQL statement executable by a Hive SQL engine; wherein the SQL statement is obtained by being customized according to the requirement of a data receiver; and responding to the data receiver based on the processing result of the log data. In this way, different SQL statements are customized according to different requirements of the data receiver, different MapReduce programs are obtained by analyzing the different SQL statements by using the Hive SQL engine, and different processing of the log data is completed by executing the different MapReduce programs. Compared with the prior art of directly developing corresponding MapReduce programs according to different processing requirements, the technical scheme of the application reduces the development cost.

Description

Technical Field

[0001] This application relates to big data technology, and more particularly to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, telecom operators have been committed to building big data cloud platforms. The internet access log data collected from these platforms has high analytical value and can be used for scenarios such as unified customer profiling, precision marketing, relationship chain mining, and public opinion consulting reports. Typically, the collected internet access log data can also undergo preprocessing such as cleaning to improve the accuracy of subsequent analysis results.

[0003] Existing technologies typically utilize the MapReduce computing model to preprocess internet browsing log data. However, since different customers may have different needs, it is often necessary to develop corresponding MapReduce programs based on specific customer requirements, resulting in high costs. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a data processing method, apparatus, device, and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] Firstly, a data processing method is provided, the method comprising:

[0007] Retrieve log data;

[0008] The log data is processed using SQL statements that the Hive SQL engine can execute; wherein, the SQL statements are customized according to the needs of the data recipient.

[0009] Based on the processing results of the log data, a response is sent to the data recipient.

[0010] In the above scheme, the log data includes a log data type and a log data body; the method further includes: determining a target processor for processing the log data body from a variety of processors based on the log data type; starting the target processor; and processing the log data using SQL statements that the Hive SQL engine can execute includes: controlling the target processor to execute the parsing result of the SQL statement by the Hive SQL engine to process the log data body.

[0011] In the above scheme, the target processor includes a main processor and multiple sub-processors; controlling the target processor to execute the parsing results of the SQL statements by the Hive SQL engine and process the log data body includes: controlling the main processor to preprocess the log data body to obtain intermediate results; controlling the multiple sub-processors to execute the parsing results of multiple SQL statements by the Hive SQL engine and to process the intermediate results in parallel.

[0012] In the above scheme, the SQL statement includes a Hive user-defined function; wherein the Hive user-defined function is used to perform at least one of the following data processing functions: encryption, fixed-length truncation, and specific encoding.

[0013] In the above scheme, responding to the data receiver based on the processing result of the log data includes: caching the processing result in a cache area; retrieving the processing result from the cache area and sending it to the data receiver.

[0014] In the above scheme, caching the processing result to the cache area includes: determining the cache area corresponding to each subprocessor in the target processor according to the first correspondence between the subprocessor and the cache area; and caching the processing result of each subprocessor to the corresponding cache area.

[0015] In the above scheme, obtaining the processing result from the cache and sending it to the data receiver includes: determining the data receiver corresponding to the cache based on the second correspondence between the cache and the data receiver; and sending the processing result of the cache to the corresponding data receiver.

[0016] In the above scheme, obtaining log data includes: obtaining the log data from the data source based on a shared data transmission protocol; responding to the data receiver includes: sending the processing result to the data receiver based on the shared data transmission protocol.

[0017] In the above scheme, when the data receiver is a cluster, sending the processing result to the data receiver includes: sending the processing result to the cluster so that the cluster stores the processing result.

[0018] Secondly, a data processing apparatus is provided, the apparatus comprising:

[0019] The acquisition unit is used to acquire log data;

[0020] The processing unit is used to process the log data using SQL statements that the Hive SQL engine can execute; wherein the SQL statements are customized according to the needs of the data recipient.

[0021] The response unit is used to respond to the data receiver based on the processing result of the log data.

[0022] Thirdly, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the aforementioned method when running the computer program.

[0023] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0024] By adopting the above technical solution, different SQL statements are customized according to the different needs of the data recipient. The HiveSQL engine is used to parse the different SQL statements to obtain different MapReduce programs. By executing different MapReduce programs, different processing of log data is completed. Compared with the existing method of directly developing corresponding MapReduce programs according to different processing requirements, the technical solution of this application reduces development costs. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the first process of the data processing method in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the second process of the data processing method in the embodiments of this application;

[0027] Figure 3 This is a flowchart illustrating the Flume data processing method based on the SDTP protocol in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the third process of the data processing method in the embodiments of this application;

[0029] Figure 5 This is a schematic diagram of the structure of the data processing device in the embodiments of this application;

[0030] Figure 6 This is a schematic diagram of the structure of the electronic equipment in the embodiments of this application. Detailed Implementation

[0031] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0032] This application provides a data processing method. Figure 1This is a schematic diagram of the first process of the data processing method in the embodiments of this application, such as... Figure 1 As shown, the data processing method may specifically include:

[0033] Step 101: Obtain log data.

[0034] It should be noted that log data refers to data generated by a computer operating system or application software during operation. For example, log data may include music log data, video log data, game log data, internet browsing log data, and voice log data.

[0035] In some embodiments, step 101 specifically includes: obtaining the log data from the data source based on a shared data transmission protocol.

[0036] Here, the data source, as the name suggests, is the origin of the data, which is the device or raw media that provides the required data. For example, the data source could be the database used by various provinces.

[0037] In practical applications, the data processing device corresponding to the data processing method of this application supports data acquisition using the Shared Data Transfer Protocol (SDTP), and therefore can obtain log data from the data source based on the SDTP protocol.

[0038] Step 102: Process the log data using SQL statements that the Hive SQL engine can execute; wherein the SQL statements are customized according to the needs of the data recipient.

[0039] It should be noted that, due to the different needs of different data recipients, log data processing must be tailored to these specific requirements. This application defines different SQL statements based on the different needs of the data recipients. These different SQL statements can be understood as processing strategies for different aspects of the log data body. By utilizing the various SQL statements that the HiveSQL engine can execute, different processing methods are applied to the log data to meet the needs of different data recipients.

[0040] Specifically, during the data processing, the Hive SQL engine is used to parse the custom SQL statement to obtain the corresponding MapReduce program, and the data processing of the log data is completed by executing the MapReduce program.

[0041] Here, the method of using the Hive SQL engine to parse SQL statements and generate MapReduce programs in this application reduces development costs compared to the existing technology of directly developing corresponding MapReduce programs based on the data receiver's processing requirements for log data bodies.

[0042] Step 103: Based on the processing result of the log data, respond to the data receiver.

[0043] For example, the processing results of log data can be sent directly to the data recipient, or indirectly to the data recipient.

[0044] For example, in some embodiments of the indirect transmission method, step 103 specifically includes: caching the processing result in a cache area; retrieving the processing result from the cache area and sending it to the data receiver.

[0045] It should be noted that in this embodiment, the processing result is cached in the cache area, and then the processing result is retrieved from the cache area and sent to the data receiver. The purpose of doing this is to avoid the possibility of data loss when the processing result is sent directly to the data receiver.

[0046] Here, after confirming that the data receiver has fully received the processing result, the processing result cached in the buffer is usually deleted, thus saving some buffer space and memory.

[0047] In some embodiments, responding to the data receiver includes: sending the processing result to the data receiver based on the shared data transmission protocol.

[0048] In practical applications, the data processing device corresponding to the data processing method of this application also supports data transmission of the SDTP protocol (i.e., shared data transmission protocol), that is, the processing result is sent to the data receiver based on the SDTP protocol.

[0049] It should be noted that in this embodiment, the data receiver can be the acquisition end (for example, the acquisition end can be a data application tenant). The processing result is sent to the acquisition end based on the SDTP protocol, and the acquisition end can perform further parsing on the processed log data.

[0050] In some embodiments, when the data receiver is a cluster, sending the processing result to the data receiver includes: sending the processing result to the cluster so that the cluster stores the processing result.

[0051] It's important to note that a cluster is a loosely coupled collection of computing nodes consisting of two or more nodes (servers) that provide a single client view of network services or applications (including databases, web services, and file services). Here, the cluster is used to provide database services to users, storing the received processing results. For example, a cluster may include at least HDFS and Kafka.

[0052] In this embodiment, the processing results obtained from the cache are sent to the cluster for storage, so that the required data type can be directly retrieved from the corresponding cluster in the future.

[0053] Here, the entity executing steps 101 to 103 can be the processor of an electronic device.

[0054] By adopting the above technical solution, different SQL statements are customized according to the different needs of the data recipient. The HiveSQL engine is used to parse the different SQL statements to obtain different MapReduce programs. By executing different MapReduce programs, different processing of log data is completed. Compared with the existing method of directly developing corresponding MapReduce programs according to different processing requirements, the technical solution of this application reduces development costs.

[0055] Based on the above embodiments, this application specifically provides a data processing method. Figure 2 This is a schematic diagram of the second process of the data processing method in the embodiments of this application.

[0056] like Figure 2 As shown, the specific steps may include:

[0057] Step 201: Obtain log data; wherein the log data includes log data type and log data body.

[0058] In some embodiments, this step specifically includes: obtaining the log data from the data source based on a shared data transmission protocol.

[0059] Step 202: Based on the log data type, determine the target processor for processing the log data body from a variety of processors.

[0060] It should be noted that different processors are used to process different types of log data bodies, and different types of log data bodies have different processing strategies; that is, different processors correspond to different processing strategies. Before performing processing operations, the target processor for processing the log data body is determined from multiple processors based on the log data type, and then the target processor is started.

[0061] For example, a mapping table between log data types and processor names can be set in advance. After the log data type of the acquired log data is determined, the target processor name corresponding to the log data type is determined from the mapping table, and then the target processor is started according to the target processor name.

[0062] Step 203: Start the target processor.

[0063] Step 204: Control the target processor to execute the parsing results of the SQL statement by the Hive SQL engine and process the log data body; wherein, the SQL statement is customized according to the needs of the data receiver.

[0064] It should be noted that, due to the different needs of different data recipients, log data processing must be tailored to these specific requirements. This application defines different SQL statements based on the different needs of the data recipients. These different SQL statements can be understood as processing strategies for different aspects of the log data body. By utilizing the various SQL statements that the HiveSQL engine can execute, different processing methods are applied to the log data to meet the needs of different data recipients.

[0065] In this embodiment, the Hive SQL engine is used to parse the custom SQL statement to obtain the corresponding MapReduce program. The target processor is then controlled to execute the MapReduce program to complete the data processing of the log data.

[0066] For example, in some embodiments, the target processor includes a main processor and multiple sub-processors; step 204 specifically includes: controlling the main processor to preprocess the log data body to obtain intermediate results; controlling the multiple sub-processors to execute the parsing results of multiple SQL statements by the Hive SQL engine, and performing parallel processing on the intermediate results.

[0067] In other words, the main processor preprocesses the log data body once to obtain an intermediate result, and then controls each subprocessor in parallel to process the intermediate result according to different requirements. Here, in this embodiment, the log data body is only preprocessed once by the main processor, but it will be processed again by all the subprocessors according to different requirements.

[0068] In some embodiments, the SQL statement includes a Hive custom function; wherein the Hive custom function is used to perform at least one of the following data processing functions: encryption, fixed-length truncation, and specific encoding.

[0069] Here, flexible data processing is achieved by setting Hive custom functions in SQL statements. Specific Hive custom functions can perform data processing such as encryption (e.g., MD5 encryption), fixed-length truncation, and specific encoding.

[0070] Step 205: Based on the processing result of the log data, respond to the data receiver.

[0071] For example, the processing results of log data can be sent directly to the data recipient, or indirectly to the data recipient.

[0072] For example, in some embodiments of the indirect transmission method, step 103 specifically includes: caching the processing result in a cache area; retrieving the processing result from the cache area and sending it to the data receiver.

[0073] It should be noted that in this embodiment, the processing result is cached in the cache area, and then the processing result is retrieved from the cache area and sent to the data receiver. The purpose of doing this is to avoid the possibility of data loss when the processing result is sent directly to the data receiver.

[0074] Here, after confirming that the data receiver has fully received the processing result, the processing result cached in the buffer is usually deleted, thus saving some buffer space and memory.

[0075] In some embodiments, caching the processing result to a cache includes: determining the cache corresponding to each subprocessor in the target processor according to a first correspondence between subprocessors and caches; and caching the processing result of each subprocessor to the corresponding cache.

[0076] In other words, a preliminary correspondence is established between subprocessors and caches. When caching processing results, the cache corresponding to each subprocessor is first determined based on the preliminary correspondence, and then the processing results of each subprocessor are cached in the corresponding cache. Different subprocessors may correspond to the same cache or different caches.

[0077] In some embodiments, obtaining the processing result from the cache and sending it to the data receiver includes: determining the data receiver corresponding to the cache based on a second correspondence between the cache and the data receiver; and sending the processing result of the cache to the corresponding data receiver.

[0078] In other words, a second correspondence is pre-established between the buffer and the data receiver. When sending the processing result, the data receiver corresponding to the buffer is first determined according to the second correspondence, and then the processing result obtained from the buffer is sent to the corresponding data receiver. Different buffers can correspond to the same data receiver or different data receivers.

[0079] In some embodiments, responding to the data receiver includes: sending the processing result to the data receiver based on the shared data transmission protocol.

[0080] In practical applications, the data processing device corresponding to the data processing method of this application also supports data transmission of the SDTP protocol (i.e., shared data transmission protocol), that is, the processing result is sent to the data receiver based on the SDTP protocol.

[0081] It should be noted that in this embodiment, the data receiver can be the acquisition end (for example, the acquisition end can be a data application tenant). The processing result is sent to the acquisition end based on the SDTP protocol, and the acquisition end can perform further parsing on the processed log data.

[0082] In some embodiments, when the data receiver is a cluster, sending the processing result to the data receiver includes: sending the processing result to the cluster so that the cluster stores the processing result.

[0083] It's important to note that a cluster is a loosely coupled collection of computing nodes consisting of two or more nodes (servers) that provide a single client view of network services or applications (including databases, web services, and file services). Here, the cluster is used to provide database services to users, storing the received processing results. For example, a cluster may include at least HDFS and Kafka.

[0084] In this embodiment, the processing results obtained from the cache are sent to the cluster for storage, so that the required data type can be directly retrieved from the corresponding cluster in the future.

[0085] Based on the above embodiments, this application differs from a single collection service. Instead, it implements one-time parsing (i.e., the main processor's preprocessing of the log data body), multi-path processing (i.e., multiple sub-processors perform multi-path processing of the preprocessed results) and output of log data during the collection process, thereby reducing data persistence and processing times.

[0086] By adopting the above technical solution, different SQL statements are customized according to the different needs of the data receiver. The HiveSQL engine parses these SQL statements to obtain different MapReduce programs. These MapReduce programs are then executed in parallel by controlling the target processor (including the main processor and multiple sub-processors) to perform different processing on the log data in parallel, achieving flexible processing of log data and improving data processing efficiency. Furthermore, compared to existing methods that directly develop corresponding MapReduce programs based on different processing requirements, this technical solution reduces development costs.

[0087] Based on the above embodiments, this application specifically provides a Flume data processing framework based on the SDTP protocol. Figure 3 This is a schematic diagram of the Flume data processing framework based on the SDTP protocol in the embodiments of this application.

[0088] like Figure 3 As shown, deployment unit 30 is the Flume Agent deployment unit, mainly used for receiving, processing, caching, and sending data. Deployment unit 30 includes a collection layer 32 (SDTPSource), a processing layer 33 (SDTPProcessor), a caching layer 34 (ChannelSelector) (including cache 1, cache 2, and cache 3), and a sending layer (including sender 1, sender 2, and sender 3). Specifically, collection layer 32 (SDTPSource) receives log data; processing layer 33 (SDTPProcessor) processes the log data; caching layer 34 (ChannelSelector) caches the processed data; and the sending layer sends the data obtained from cache layer 34 (ChannelSelector) to the corresponding data receivers (including receiver 1, receiver 2, and receiver 3).

[0089] based on Figure 3 The data processing framework shown presents a data processing method. Figure 4 This is a schematic diagram of the third process of the data processing method in the embodiments of this application, such as... Figure 4 As shown, the data processing steps may specifically include:

[0090] Step 401: SDTPSource obtains log data from the data source based on the SDTP protocol.

[0091] Here, SDTPSource obtains log data forwarded by the provincial side (i.e., the data source) via the reporting gateway 31 through the SDTP protocol. This log data can also be referred to as XDR data.

[0092] Before acquiring log data, it is usually necessary to configure the corresponding configuration information (such as data source address information and data type information) according to the recipient's requirements in order to collect the corresponding log data. Below is a specific example of the configuration information.

[0093] Step 402: Based on the log data type, SDTPSource determines the s1u processor from a variety of processors to process the log data body; wherein, the log data includes the log data type and the log data body.

[0094] Here, SDTPSource also needs to forward the acquired log data to SDTPProcessor for processing. Since different master processors of SDTPProcessor handle log data bodies of different log data types, it is necessary to match the corresponding master processor for the current log data type in order to send the log data body to the corresponding master processor for preprocessing. To this end, the SDTPSource configuration information sets the file path that stores the mapping relationship between log data types (i.e., XDRType in the example below) and master processors (i.e., the router.confPath configuration in the example below, pointing to a local file path). Subsequently, accessing this file path will determine the master processor that processes the log data body.

[0095] For example, SDTPSource configuration information may include:

[0096] #sdtp source binds to the IP address and port

[0097] agent.sources.src1.bind=10.154.6.3

[0098] agent.sources.src1.port=9100

[0099] #type of source

[0100] agent.sources.src1.type=flume.sdtp.SDTPSource

[0101] #source adds extra tags to the processed data.

[0102] agent.sources.src1.extra.header=province:JS

[0103] #source uses the private key file and server certificate.

[0104] agent.sources.src1.key.file.path= / home / flume / certs / server.key

[0105] agent.sources.src1.cert.file.path= / home / flume / certs / server.crt

[0106] agent.sources.src1.ca.file.path= / home / flume / certs / ca.crt

[0107] The router file used by #source represents the mapping relationship between XDRType and the main processor.

[0108] agent.sources.src1.router.confPath= / home / flume / router.conf

[0109] The following format specifies the mapping relationship between XDR Type and the main processor. When XDR Type is determined to be 2, the main processor is determined to be s1u; when XDR Type is determined to be 201, the main processor is determined to be mc_voice.

[0110] Router.conf configuration examples include:

[0111] 2:s1u

[0112] 201:mc_voice

[0113] Step 403: The SDTPProcessor controls the s1u processor to preprocess the log data body and obtain intermediate results.

[0114] For example, the configuration information of SDTPProcessor for the s1u processor may include:

[0115] #source is a list of built-in processors, which can contain multiple processors separated by commas.

[0116] agent.sources.src1.processors=s1u

[0117] The list of subprocessors for the #s1u processor, which can contain multiple subprocessors.

[0118] agent.sources.src1.processor.s1u.processors=hdfs,kafka

[0119] #Whether to check for data latency agent.sources.src1.processor.s1u.latencyInspector.on = true

[0120] #s1u processes fixed-length data

[0121] agent.sources.src1.processor.s1u.input.column.lengths=2,2,2,2,2,1,1,16,1,8,8,16,1,8,8,8,8,1,1,1,1,1,1,1,1,1,1,1,4,2,1,4,1,2,1,4,4,4,16,16,16,

[0122] The S1U processor can be used to check the timeliness of data and to process fixed-length data (as shown in the configuration information above) and symbol-separated values ​​(CSV).

[0123] The format definition of fixed-length data is specified by input.column.lengths in the S1U processor. Each column is assigned a length. For example, the special field 1+

[16] *N consists of a 1-byte header and 16*N bytes, which adapts to the logic of variable field definitions in the data. If it is a CSV format, the input.column.num property needs to be defined to configure the number of fields, and the input.column.delimiter property needs to be defined to configure the field delimiter.

[0124] Here, log data will only be processed once by the s1u processor, but it will be processed separately by all the sub-processors in the s1u processor (including the HDFS sub-processor and the Kafka sub-processor).

[0125] Step 404: The SDTPProcessor controls the three subprocessors in the s1u processor to execute the parsing results of the three SQL statements by the Hive SQL engine, and performs parallel processing on the intermediate results to obtain three processing results;

[0126] Here, we will take two specific examples of the S1U processor: the HDFS subprocessor and the Kafka subprocessor.

[0127] For example, the configuration information of the SDTPProcessor for the HDFS subprocessor and Kafka subprocessor in the S1U processor includes:

[0128] #HDFS Subprocessor

[0129] agent.sources.src1.processor.s1u.processor.hdfs.serializer.type=txt

[0130] agent.sources.src1.processor.s1u.processor.hdfs.serializer.maxRecordNum=1

[0131] agent.sources.src1.processor.s1u.processor.hdfs.serializer.delimiter=|

[0132] #SQL statement

[0133] agent.sources.src1.processor.s1u.processor.hdfs.sql=select c1,tbcd(c2),tbcd(c3),

[0134] tbcd(c4),tbcd(c5),c6,c7,c8,c9,tbcd(c10),tbcd(c11),sm4bytes(tbcd(c12),

[0135] bytestotimestamp(c14)),c13,bytestotimestamp(c14),bytestotimestamp(c15),

[0136] bytestodouble(c16),bytestodouble(c17),c18,c19,c20,c21,c22,c23,c24,c25,

[0137] c26,c27,c28,c29,c30,c31,c32,c33,c34,c35,c36,c37,bytestoipv4(c38),

[0138] bytestoipv6(c39),bytestoip(c40),bytestoip(c41),c42,c43,c44,c45,c46,c47,c48,

[0139] BearerDecodes1u220(c49) from event where c1 is not null and c12 is not null

[0140] #Kafka Subprocessor

[0141] agent.sources.src1.processor.mme.processor.kafka.serializer.maxRecordNum=1000

[0142] agent.sources.src1.processor.mme.processor.kafka.serializer.type=txt

[0143] agent.sources.src1.processor.mme.processor.kafka.serializer.delimiter=|

[0144] #SQL statement

[0145] agent.sources.src1.processor.mme.processor.kafka.sql=select c9,null(),c2,null(),null(),sm4bytes(tbcd(c6),c10),sm4bytes(normalize(tbcd(c8)),c10),sm4bytes(tb cd(c7),c10),c10,null(),null(),null(),c33,c34,subbytes(tbcd(c6),0,3),subbytes(normaliz e(tbcd(c8)),0,7),subbytes(tbcd(c7),0,6)from eventwhere c1 is not null

[0146] The HDFS subprocessor checks whether the log data is in TXT file format, whether the maximum number of records is 1, and whether the field separator is "|". The HDFS subprocessor also includes custom SQL statements, primarily used to add data processing functionality.

[0147] The Kafka subprocessor checks whether the log data file format is TXT, whether the maximum number of records is 1000, and whether the field separator is "|". The Kafka subprocessor also includes custom SQL statements, primarily used to add data processing functionality.

[0148] Here, the method of using the Hive SQL engine to parse SQL statements and generate MapReduce jobs, as described in this application, reduces development costs compared to directly developing corresponding MapReduce programs based on the data receiver's processing requirements for log data bodies.

[0149] Step 405: SDTPProcessor caches the three processing results into three cache areas;

[0150] Based on this step, ChannelSelector needs to have its cache information pre-configured. For example, ChannelSelector configuration information includes:

[0151] #Set selector type

[0152] agent.sources.src1.selector.type=multiplexing

[0153] agent.sources.src1.selector.header=target

[0154] # Define mapping tags for different channels

[0155] agent.sources.src1.selector.mapping.ch1 = ch1 (i.e., cache 1)

[0156] agent.sources.src1.selector.mapping.ch2 = ch2 (i.e., cache 2)

[0157] In addition, the SDTPProcessor configuration information also includes:

[0158] # Mark which channel the result will be output to in the processor

[0159] agent.sources.src2.processor.ps_gen.processor.hdfs.extra.header=target:ch1

[0160] agent.sources.src2.processor.ps_gen.processor.kafka.extra.header=target:ch2

[0161] The processing results of the HDFS subprocessor are cached in ch1 (i.e., cache 1), and the processing results of the Kafka subprocessor are cached in ch2 (i.e., cache 2). Configuration information for cache 3 is not provided here.

[0162] Step 406: The sending layer retrieves three processing results from three buffers and sends them in parallel to three data receivers.

[0163] Here, the sending layer includes three senders: HDFSSinks, KafkaSinks, and SDTPSinks. HDFSSinks retrieves the processing results from cache 1 and sends them to the HDFS cluster (receiver 1) for storage. KafkaSinks retrieves the processing results from cache 2 and sends them to the Kafka cluster (receiver 2) for storage. SDTPSinks retrieves the processing results from cache 3 and sends them to the next collector (receiver 3) based on the SDTP protocol.

[0164] Here, the address information and data transmission method of receiver 3 need to be configured in advance in SDTPSinks.

[0165] Based on the data processing method demonstrated in the above embodiments, the data processed by SDTPProcessor can be sent to HDFS, Kafka storage, and SDTPsink to achieve multi-channel output. Unlike a single collection service, this application does not require additional data processing from Kafka or Storm clusters. During the collection process, it implements optimized designs such as multi-version data configuration, custom SQL parsing and processing, and SDTP protocol plugins, forming a Flume plugin that supports SDTP protocol data transmission. This plugin can be applied to online DPI data collection systems, completing internet access log data collection in 31 provinces, with a daily data volume exceeding 550TB.

[0166] By adopting the above technical solution, different SQL statements are customized according to the different needs of the data receiver. The HiveSQL engine parses these SQL statements to obtain different MapReduce programs. These MapReduce programs are then executed in parallel by controlling the target processor (including the main processor and multiple sub-processors) to perform different processing on the log data in parallel, achieving flexible processing of log data and improving data processing efficiency. Furthermore, compared to existing methods that directly develop corresponding MapReduce programs based on different processing requirements, this technical solution reduces development costs.

[0167] To implement the method of the embodiments of this application, based on the same inventive concept, a data processing apparatus is also provided in the embodiments of this application. Figure 5This is a schematic diagram of the structure of the data processing device in the embodiments of this application, such as... Figure 5 As shown, the data processing apparatus includes:

[0168] Acquisition unit 501 is used to acquire log data;

[0169] The processing unit 502 is used to process the log data using SQL statements that the Hive SQL engine can execute; wherein the SQL statements are customized according to the needs of the data receiver.

[0170] The response unit 503 is used to respond to the data receiver based on the processing result of the log data.

[0171] By adopting the above technical solution, different SQL statements are customized according to the different needs of the data recipient. The HiveSQL engine is used to parse the different SQL statements to obtain different MapReduce programs. By executing different MapReduce programs, different processing of log data is completed. Compared with the existing method of directly developing corresponding MapReduce programs according to different processing requirements, the technical solution of this application reduces development costs.

[0172] In some embodiments, the log data includes a log data type and a log data body; based on the log data type, a target processor for processing the log data body is determined from a variety of processors; the target processor is started; the processing unit 502 is specifically used to control the target processor to execute the parsing result of the SQL statement by the Hive SQL engine to process the log data body.

[0173] In some embodiments, the target processor includes a main processor and multiple sub-processors; controlling the target processor to execute the parsing results of the SQL statements by the Hive SQL engine and process the log data body includes: controlling the main processor to preprocess the log data body to obtain intermediate results; controlling the multiple sub-processors to execute the parsing results of multiple SQL statements by the Hive SQL engine and to process the intermediate results in parallel.

[0174] In some embodiments, the SQL statement includes a Hive custom function; wherein the Hive custom function is used to perform at least one of the following data processing functions: encryption, fixed-length truncation, and specific encoding.

[0175] In some embodiments, the response unit 503 is specifically configured to cache the processing result in a cache area; retrieve the processing result from the cache area; and send it to the data receiver.

[0176] In some embodiments, caching the processing result to a cache includes: determining the cache corresponding to each subprocessor in the target processor according to a first correspondence between subprocessors and caches; and caching the processing result of each subprocessor to the corresponding cache.

[0177] In some embodiments, obtaining the processing result from the cache and sending it to the data receiver includes: determining the data receiver corresponding to the cache based on a second correspondence between the cache and the data receiver; and sending the processing result of the cache to the corresponding data receiver.

[0178] In some embodiments, the acquisition unit 501 is specifically used to acquire the log data from the data source based on a shared data transmission protocol; when responding to the data receiver, it includes: sending the processing result to the data receiver based on the shared data transmission protocol.

[0179] In some embodiments, when the data receiver is a cluster, sending the processing result to the data receiver includes: sending the processing result to the cluster so that the cluster stores the processing result.

[0180] This application also provides another electronic device. Figure 6 This is a schematic diagram of the structure of the electronic devices in the embodiments of this application, such as... Figure 6 As shown, the electronic device includes: a processor 601 and a memory 602 configured to store computer programs capable of running on the processor;

[0181] The processor 601 is configured to execute the method steps in the foregoing embodiments when running a computer program.

[0182] Of course, in practical applications, such as Figure 6 As shown, the various components in this electronic device are coupled together via a bus system 603. It is understood that the bus system 603 is used to implement communication between these components. In addition to a data bus, the bus system 603 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 603.

[0183] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the aforementioned processor can also be other types, and the embodiments of this application do not specifically limit this.

[0184] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0185] In an exemplary embodiment, this application also provides a computer-readable storage medium for storing a computer program.

[0186] Optionally, the computer-readable storage medium can be applied to any of the methods in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the processor in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0188] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0191] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0192] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0193] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve log data; The log data is processed using SQL statements that the Hive SQL engine can execute; wherein the SQL statements are customized according to the needs of the data recipient; and wherein the Hive SQL engine is used to parse different SQL statements to obtain different MapReduce programs. Based on the processing results of the log data, a response is made to the data recipient; The process of processing the log data using SQL statements that can be executed by the Hive SQL engine includes: controlling the main processor in the target processor to preprocess the log data body in the log data to obtain intermediate results; and controlling multiple sub-processors in the target processor to execute the parsing results of multiple SQL statements by the Hive SQL engine to process the intermediate results in parallel.

2. The method according to claim 1, characterized in that, The log data also includes log data types; The method further includes: determining a target processor for processing the log data body from a variety of processors based on the log data type.

3. The method according to claim 1, characterized in that, The SQL statement includes Hive user-defined functions; The Hive custom function is used to perform at least one of the following data processing functions: encryption, fixed-length truncation, and specific encoding.

4. The method according to claim 1, characterized in that, The step of responding to the data recipient based on the processing result of the log data includes: The processing result is cached in the cache area; The processing result is obtained from the cache and sent to the data receiver.

5. The method according to claim 4, characterized in that, The step of caching the processing result to the cache area includes: Based on the first correspondence between subprocessors and cache areas, determine the cache area corresponding to each subprocessor in the target processor; The processing results of each subprocessor are cached in the corresponding cache area.

6. The method according to claim 4, characterized in that, The step of retrieving the processing result from the cache and sending it to the data receiver includes: The data receiver corresponding to the cache is determined based on the second correspondence between the cache and the data receiver; The processing results of the buffer are sent to the corresponding data receiver.

7. The method according to claim 1, characterized in that, The acquisition of log data includes: The log data is obtained from the data source based on a shared data transmission protocol; The response to the data receiver includes: Based on the shared data transmission protocol, the processing result is sent to the data receiver.

8. The method according to claim 7, characterized in that, When the data receiver is a cluster, sending the processing result to the data receiver includes: The processing result is sent to the cluster so that the cluster stores the processing result.

9. A data processing apparatus, characterized in that, The apparatus is used to perform the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 8.

Images