Data processing method, device and system

Abstract

The present invention discloses a data processing method, device and system, and relates to the field of computer technology. The method comprises: after receiving a data report message sent by a collection client, verifying the data report message; when the data report message passes the verification, obtaining the corresponding channel information according to the data report message, and judging whether to limit the flow of the channel in the message middleware according to the channel information; when there is no need to limit the flow of the channel, extracting the collected terminal data from the data report message, and writing the terminal data into the channel. Through the above steps, the unified collection and output of multi-terminal data is realized, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device and system. Background Art

[0002] Before conducting data analysis, the first thing to solve is the data collection problem. Generally speaking, the data that needs to be collected can be roughly divided into the following types: the first type is user behavior data on web pages. The characteristics of this type of data are large data volume and scattered reporting addresses; the second type is real-time log data on the server. The characteristics of this type of data are moderate data volume and concentrated reporting addresses; the third type is real-time data uploaded by users. The characteristics of this type of data are small data volume, but the data format is flexible.

[0003] In the prior art, there are different data collection schemes for different data. For the first type of data, data collection is currently mostly carried out by front-end code embedding, that is, developers use front-end code (such as JavaScript code) to monitor control events on the Web page, collect user behavior data when the front-end code monitors the control events triggered by the user, and send it to the back-end server; for the latter two types of data, data collection is currently mostly based on the HTTP interface provided by the data collection server, that is, the user encapsulates the data to be reported into an HTTP request, and reports the data by sending an HTTP request to the back-end server.

[0004] In the process of implementing the present invention, the inventor of the present invention found that the collection solutions for different terminal data in the prior art are too scattered and customized; for data analysts who need these data, they need to access different data output sources to obtain data from different terminals, resulting in high access costs and low efficiency. Summary of the invention

[0005] In view of this, the present invention provides a data processing method, device and system, which realize the unified collection and output of multi-terminal data, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of data output source and improves the access efficiency of data output source.

[0006] To achieve the above objective, according to a first aspect of the present invention, a data processing method is provided.

[0007] The data processing method of the present invention comprises: after receiving a data reporting message sent by a collection client, verifying the data reporting message; wherein, there are multiple collection clients for collecting data from multiple terminals; when the data reporting message passes the verification, obtaining corresponding channel information according to the data reporting message, and judging whether to limit the flow of the channel in the message middleware according to the channel information; when there is no need to limit the flow of the channel, extracting the collected terminal data from the data reporting message, and writing the terminal data into the channel.

[0008] Optionally, the verifying the data reporting message includes: verifying the correctness of the data according to a verification code carried by the data reporting message, and / or, performing a field integrity check on the data reporting message, and / or, verifying the user identity according to the user information carried by the data reporting message.

[0009] Optionally, obtaining corresponding channel information according to the data reporting message, and determining whether to limit the flow of the channel in the message middleware according to the channel information includes: determining the corresponding channel identifier according to the application identifier carried by the data reporting message; querying the corresponding channel information according to the channel identifier; wherein the channel information includes a flow limiting flag of the channel; if the flow limiting flag of the channel is a first value, there is no need to limit the flow of the channel in the message middleware; if the flow limiting flag of the channel is a second value, it is necessary to limit the flow of the channel in the message middleware.

[0010] Optionally, the channel information also includes: a time period, and the amount of data stored in the channel within the time period; the method also includes: after writing the terminal data into the channel, updating the amount of data stored in the channel within the time period according to the amount of data of the terminal data; when the updated amount of data stored in the channel is greater than the current limiting threshold, setting the current limiting flag of the channel to a second value.

[0011] Optionally, the amount of data stored in the channel within the time period includes: the number of data items stored in the channel within the time period, and the number of bytes of data stored in the channel within the time period; the situation where the updated amount of data stored in the channel is greater than the current limiting threshold includes: the number of data items stored in the channel after the update is greater than the first current limiting threshold, or the number of bytes of data stored in the channel after the update is greater than the second current limiting threshold.

[0012] Optionally, extracting the collected terminal data from the data reporting message and writing the terminal data into the channel includes: extracting the collected terminal data from the data reporting message; reading data structure configuration information corresponding to the channel; assembling the terminal data according to the data structure configuration information, and writing the assembled data into the channel.

[0013] Optionally, the method further includes: when it is necessary to limit the flow of the channel, writing the data reporting message into a cache.

[0014] To achieve the above objective, according to a second aspect of the present invention, a data processing device is provided.

[0015] The data processing device of the present invention comprises: a verification module, which is used to verify the data reporting message after receiving the data reporting message sent by the collection client; wherein there are multiple collection clients, which are used to collect data from multiple terminals;

[0016] A judgment module, used for obtaining corresponding channel information according to the data reporting message when the data reporting message passes the verification, and judging whether to limit the flow of the channel in the message middleware according to the channel information;

[0017] The writing module is used to extract the collected terminal data from the data reporting message and write the terminal data into the channel without limiting the current of the channel.

[0018] To achieve the above objective, according to a third aspect of the present invention, a data processing system is provided.

[0019] The data processing system of the present invention includes: multiple collection clients, a gateway, and a message middleware; the multiple collection clients are deployed on multiple terminals and are used to collect data from the multiple terminals; the gateway is provided with a data processing device, which is used to verify the data reporting message after receiving the data reporting message sent by any one of the multiple collection clients; it is also used to obtain the corresponding channel information according to the data reporting message when the data reporting message passes the verification, and judge whether to limit the flow of the channel in the message middleware according to the channel information; it is also used to extract the collected terminal data from the data reporting message and write the terminal data into the channel when there is no need to limit the flow of the channel; the message middleware is used to output the terminal data written into the channel.

[0020] To achieve the above objective, according to a fourth aspect of the present invention, an electronic device is provided.

[0021] The electronic device of the present invention comprises: one or more processors; and a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method of the present invention.

[0022] To achieve the above objective, according to a fifth aspect of the present invention, a computer readable medium is provided.

[0023] The computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, the data processing method of the present invention is implemented.

[0024] An embodiment of the above invention has the following advantages or beneficial effects: after receiving the data reporting message sent by the collection client, the data reporting message is verified, and the corresponding channel information is obtained according to the data reporting message after the data reporting message passes the verification, and it is determined whether to limit the flow of the channel in the message middleware according to the channel information, and when no flow limiting is required, the collected terminal data is extracted from the data reporting message and the terminal data is written into the channel. These steps realize the unified collection and output of multi-terminal data, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0027] Figure 1 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0028] Figure 2 is a schematic diagram of the main flow of a data processing method according to a first embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of main modules of a data processing device according to a third embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of main modules of a data processing system according to a fourth embodiment of the present invention;

[0032] Figure 6It is a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] It should be pointed out that the various embodiments of the present invention and the technical features in the embodiments may be combined with each other without affecting the implementation of the present invention.

[0035] Before describing the embodiments of the present invention in detail, some technical terms involved in the embodiments of the present invention are first described.

[0036] Kafka: An open source stream processing platform developed by the Apache Software Foundation, written in Scala and Java. The goal of the project is to provide a unified, high-throughput, low-latency platform for processing real-time data.

[0037] Topic: The purpose of introducing Topics in Kafka is to classify messages. When saving messages in Kafka, they are classified by Topic.

[0038] Flume: is a distributed, highly reliable, and scalable data collection service that provides customized functions.

[0039] HTTP Client: It is a sub-project under Apache Jakarta Common, which can be used to provide an efficient, up-to-date, and feature-rich client programming toolkit that supports the HTTP protocol. It also supports the latest versions and recommendations of the HTTP protocol.

[0040] Figure 1 An exemplary system architecture 100 is shown to which a data processing method or a data processing device according to an embodiment of the present invention can be applied.

[0041] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0042] The terminal devices 101 , 102 , 103 interact with the server 105 via the network 104 to receive or send messages, etc. The terminal devices 101 , 102 , 103 are provided with a collection client for collecting terminal data and reporting the collected terminal data to the server 105 via the network 104 .

[0043] The terminal devices 101, 102, 103 may be, for example, various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. In addition, the terminal devices 101, 102, 103 may also be background servers.

[0044] The server 105 is a background management server that uniformly processes and outputs the data collected from the terminal devices 101, 102, and 103. After receiving the data report message from the terminal device, the background management server can process the data report message and return the processing result to the terminal device.

[0045] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 105 , and accordingly, the data processing device is generally disposed in the server 105 .

[0046] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0047] Figure 2 FIG. 1 is a schematic diagram of the main flow of the data processing method according to the first embodiment of the present invention. Figure 2 As shown, the data processing method of the embodiment of the present invention includes:

[0048] Step S201: after receiving the data reporting message sent by the acquisition client, verify the data reporting message.

[0049] There are multiple collection clients for collecting data from multiple terminals. For example, when it is necessary to collect data from terminal 1 (such as user behavior data on a Web page) and data from terminal 2 (such as log data from a backend server), a corresponding collection client can be deployed for terminal 1 and a corresponding collection client can be deployed for terminal 2 to collect data from multiple terminals. Afterwards, the collection client can carry the collected terminal data in a data reporting message and send it to the data processing device.

[0050] In an optional implementation, verifying the data report message includes: verifying the correctness of the data according to the verification code carried by the data report message. For example, the verification code carried by the data report message can be compared with a pre-stored verification code or a verification code generated in real time; if the two are consistent, it is confirmed that the data carried by the message has not been tampered with, that is, the data carried by the message is correct; if the two are inconsistent, it is confirmed that the data carried by the message has been tampered with, that is, the data carried by the message is incorrect. Through the above steps, the security of the data can be improved.

[0051] In another optional implementation, verifying the data report message includes: performing field integrity verification on the data report message. For example, each field in the data report message may be traversed, and if a field is missing, it is confirmed that the message has not passed the field integrity verification; if no field is missing, it is confirmed that the message has passed the field integrity verification.

[0052] In another optional implementation, verifying the data report message includes: verifying the user identity according to the user information carried in the data report message. For example, the user's account and password can be obtained from the data report message, and if the user's account and password match, the user is confirmed to be a legitimate user; if the user's account and password do not match, the user is confirmed to be an illegal user.

[0053] To further improve the security of data, any two or three of the above three optional implementations may be combined. In addition, in specific implementation, other methods may be used to verify the data reporting message.

[0054] Step S202: when the data reporting message passes the verification, the corresponding communication information is obtained according to the data reporting message, and it is determined whether to limit the flow of the channel in the message middleware according to the channel information.

[0055] Among them, the message middleware includes multiple channels (or message pipes), and different channels can be used to store different types of data. Exemplarily, if the message middleware adopts Kafka, the multiple channels can be multiple Topics in Kafka, and different Topics can be used to store different types of data. For example, Topic1 is used to store user behavior data on Web pages, and Topic2 is used to store log data of background servers. For another example, Topic1 is used to store data collected from application 1, and Topic2 is used to store data collected from application 2.

[0056] Exemplarily, in step S202, the channel used to store the data source data can be determined based on the data source identifier (such as the application name) carried in the data reporting message, and then the channel information is obtained, and it is determined whether the channel needs to be limited at present based on the channel information.

[0057] In an optional embodiment, the channel information includes a current limiting flag. Among them, the current limiting flag has two possible values: a first value and a second value. For example, the first value is "no" and the second value is "yes", or the first value is "0" and the second value is "1". In this optional embodiment, it can be determined whether the channel needs to be current limited by reading the value of the current limiting flag of the channel. If the current limiting flag of the channel is the first value, there is no need to current limit the channel; if the current limiting flag of the channel is the second value, it is necessary to current limit the channel.

[0058] In another optional implementation, the channel information includes the amount of data stored in the channel. In this optional implementation, the sum of the amount of terminal data carried by the data reporting message and the amount of data stored in the channel can be calculated; if the sum of the two is greater than a preset current limiting threshold, the channel needs to be current limited; if the sum of the two is less than or equal to the preset current limiting threshold, the channel does not need to be current limited.

[0059] Step S203: extracting the collected terminal data from the data reporting message and writing the terminal data into the channel without current limiting the channel.

[0060] For example, assuming that the channel corresponding to the data reporting request is Topic1, the terminal data extracted from the data reporting message is written into Topic1.

[0061] In the embodiment of the present invention, by collecting multi-terminal data based on multiple collection clients, and after receiving the data reporting message sent by the collection client, verifying the data reporting message, and judging whether the corresponding channel needs to be limited after the verification, and extracting the collected terminal data from the data reporting message when no current limiting is required, and writing the terminal data into the channel, these steps realize the unified collection and output of multi-terminal data, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

[0062] Figure 3 FIG. 2 is a schematic diagram of the main flow of the data processing method according to the second embodiment of the present invention. Figure 3 As shown, the data processing method of the embodiment of the present invention includes:

[0063] Step S301: The terminal data is collected and packaged through the collection client, and a data reporting message carrying the packaged data is sent to the gateway.

[0064] There are multiple collection clients for collecting data from multiple terminals. For example, when it is necessary to collect data from terminal 1 (such as user behavior data on a web page) and terminal 2 (such as log data from a backend server), a corresponding collection client can be deployed for terminal 1 and a corresponding collection client can be deployed for terminal 2 to collect data from multiple terminals. Afterwards, the collection client can carry the collected terminal data in a data reporting message and send it to the gateway.

[0065] For different types of terminal data, different collection clients can be used to collect data. For example, for user behavior data on a web page, data can be collected through JS SDK (a data collection point tracking tool written in JavaScript); for log data on a backend server, data can be collected through a custom Flume.

[0066] In an optional example, collecting user behavior data on a web page through the JS SDK mainly includes the following steps:

[0067] Step a1: Request the configuration center for the pre-saved tracking information, where the request input parameter is the URL (Uniform Resource Locator, also called web page address) of the web page.

[0068] Step a2: After receiving the tracking point information returned by the configuration center, the tracking point information is parsed, and the Xpath (path in XML) of the control therein is converted into CSSpath (path in CSS).

[0069] Step a3: Use CSSpath to locate the corresponding control element on the page, and bind the listening event in the tracking point information to the control element.

[0070] Step a4: After the terminal user triggers the monitoring event of the corresponding control element when accessing the Web page, data collection is performed in sequence according to the data items defined in the embedding configuration information.

[0071] After all user behavior data is collected through steps a1 to a4, the JS SDK encapsulates the data. Specifically, the collected data items can be encapsulated into a Json string in the form of Key-Value (key-value pairs), and then the data can be sent to the gateway through the network by calling HTTP Client.

[0072] Before collecting user behavior data on a web page through the JS SDK, you need to configure tracking information. In the above optional example, you can use the JS SDK to visualize tracking, which includes the following steps:

[0073] Step b1: In response to the user's trigger operation, enter the visual tracking mode.

[0074] In step b1, the user first enters a URL (webpage address) and adds an identification parameter at the end of the URL, such as the identification parameter "visualization = true". When JS-SDK recognizes the input parameter "visualization = true", it automatically enters the visualization tracking mode.

[0075] Step b2: JS-SDK automatically scans all controls on the page and binds JavaScript mouse hover events to all controls that can be embedded.

[0076] After completing step 2, all embeddable controls are bound to special trigger events.

[0077] Step b3: When the user hovers the mouse over the control, JS-SDK generates a semi-transparent mask layer on the upper layer of the control to indicate that this control can be used for visual tracking.

[0078] Step b4: When the user clicks on the mask layer, JS-SDK dynamically generates a selection box for the user to select the listening event to be bound and the data to be collected and uploaded.

[0079] Step b5: After the user saves the tracking point information, JS-SDK saves the tracking point configuration information to the configuration center on the background server.

[0080] Exemplarily, the tracking point configuration information may include: the URL of the web page corresponding to the visual tracking point, the Xpath of the tracking point control (the path in XML, used to uniquely identify the location of a control on the web page), the listening events that need to be bound, and the data items that need to be collected and reported.

[0081] For the log data on the background server, data collection can be performed through a customized Flume. Flume mainly includes Source module, Channel module and Sink module. The present invention rewrites the Sink module in Flume and customizes and implements HTTP Client in the Sink module. In specific implementation, when customizing the Sink module, it is necessary to inherit AbstractSink and rewrite the configure function and process function.

[0082] Among them, the Source module is used to receive data from the real data source, and then send the data to one or more Channel modules; the Channel module is a buffer component between the Source module and the Sink module, which is used to ensure that the Source module and the Sink module with different acquisition transmission rates can run collaboratively to ensure that the collected data is not lost; the Sink module is used for data encapsulation and sends the encapsulated data to the gateway.

[0083] Exemplarily, the custom Sink module can be used to perform the following steps: Step c1: receiving log data from the Channel module; Step c2: cleaning and filtering the log data; Step c3: encapsulating the data; Step c4: reporting streaming data through HTTP Client. In this step, the data can be sent to the gateway through the network by calling HTTP Client.

[0084] Step S302: verifying the data reporting message through the gateway.

[0085] In the embodiment of the present invention, the gateway is provided with a data processing device for executing the processing flow including step S302 to step S306.

[0086] In an optional example, the data reporting message received by the gateway includes the data items shown in Table 1:

[0087] Table 1

[0088]

[0089] In this optional example, verifying the data reporting message includes:

[0090] Step d1: Verify the correctness of the data through the verification code in the data reporting message to prevent the data from being modified during transmission.

[0091] If step d1 is verified successfully, execute step d2; if step d1 is verified failed, end the verification.

[0092] Step d2: Perform field integrity check on the data reporting message.

[0093] If a field is missing, it indicates that step d2 verification has failed, and the verification is terminated; otherwise, it indicates that step d2 verification has passed, and step d3 is executed.

[0094] Step d3: Verify based on the user identity identifier and user identity password.

[0095] Exemplarily, the user identity identifier may be a user account number, a user's mobile phone number, etc., and the user identity password may be a password set by the user. In this step, it is possible to verify whether the user identity identifier and the user identity password match. If the two match, it indicates that the user is a legitimate user, and the verification passes; if the two do not match, the verification fails.

[0096] If the data reporting message passes the verification, execute step S303; if the data reporting message fails the verification, execute step S304.

[0097] Step S303: acquiring corresponding channel information according to the data reporting message, and determining whether to limit the flow of the channel in the message middleware according to the channel information.

[0098] Among them, the message middleware includes multiple channels (or message pipes), and different channels can be used to store different types of data. Exemplarily, if the message middleware adopts Kafka, the multiple channels can be multiple Topics in Kafka, and different Topics can be used to store different types of data. For example, Topic1 is used to store user behavior data on Web pages, and Topic2 is used to store log data of background servers. For another example, Topic1 is used to store data collected from application 1, and Topic2 is used to store data collected from application 2.

[0099] In an optional example, step S303 specifically includes:

[0100] Step e1: Determine the corresponding channel identifier according to the application identifier carried in the data reporting message.

[0101] In this step, the pre-stored correspondence table between applications and channels can be queried according to the application identifier carried in the data report message to determine the corresponding channel identifier. For example, assuming that the message middleware is Kafka, the Topic used to store the application data is queried as "Topic1" according to the application identifier "app1" carried in the data report message.

[0102] Step e2: query the corresponding channel information according to the channel identifier.

[0103] The gateway is usually deployed in a distributed manner. A data report message will be randomly uploaded to any gateway server in the cluster according to the load balancing strategy. Therefore, a distributed cache (such as Redis) can be used to store channel messages.

[0104] In step e2, the distributed cache (such as Redis) can be queried according to the channel identifier to query the corresponding channel information. Exemplarily, the channel information includes: the current limiting flag of the channel, the time period, and the amount of data stored in the channel during the time period. Among them, the current limiting flag has two possible values: a first value and a second value. For example, the first value is "no" and the second value is "yes", or the first value is "0" and the second value is "1", and so on. Among them, the time period is specifically the time period of the current time window; the amount of data stored in the channel during the time period may further include: the number of data stored in the channel during the time period, and the number of bytes of data stored in the channel during the time period.

[0105] Step e3: If the current limiting flag of the channel is the first value, there is no need to limit the current of the channel in the message middleware; if the current limiting flag of the channel is the second value, it is necessary to limit the current of the channel in the message middleware.

[0106] If it is determined through steps e1 to e3 that the current does not need to be limited for the channel, step S305 is executed; if it is determined that the current needs to be limited for the channel, step S306 is executed.

[0107] Step S304: discard the data reporting message.

[0108] After the data report message verification fails in step S302, the data report message is discarded. Further, in order to improve the user experience during the data collection process, after the data report message verification fails, a verification failure prompt message may be sent to the collection client.

[0109] Step S305: extracting the collected terminal data from the data reporting message, and writing the terminal data into the channel.

[0110] Among them, step S305 may specifically include: extracting the collected terminal data from the data reporting message; reading the data structure configuration information corresponding to the channel; assembling the terminal data according to the data structure configuration information, and writing the assembled data into the channel.

[0111] For example, assuming that Kafka is used as the message middleware, the corresponding Topic can be determined based on the application identifier carried in the data report message, and the data structure configuration information corresponding to the Topic can be read from the configuration center. In addition, the collected terminal data (such as the data shown in Table 1) is extracted from the data report message. After the terminal data is assembled and processed according to the data structure configuration information, the assembled data is written to the Topic in Kafka, thereby completing the entire data collection and reporting process. Next, output empowerment can be performed through Kafka for access and use by back-end data analysts.

[0112] Furthermore, the method of the embodiment of the present invention may also include the following steps: after writing the terminal data into the channel, updating the amount of data stored in the channel within the time period according to the amount of data of the terminal data; when the amount of data stored in the channel after the update is greater than the current limiting threshold, setting the current limiting flag of the channel to a second value. After the current limiting flag is set to the second value, subsequent messages that need to be stored in the channel will be limited until the current window expires and enters the next time window.

[0113] In an optional example, the amount of data stored in the channel during the time period includes: the number of data stored in the channel during the time period, and the number of bytes of data stored in the channel during the time period. In this optional example, the sum of the number of data items of the terminal data carried by the data report message and the number of data items stored in the channel (i.e., the number of data items stored in the updated channel) can be calculated first. If the sum of the two is greater than the first current limiting threshold, the current limiting flag of the channel is set to the second value; if the sum of the two is less than or equal to the first current limiting threshold, the sum of the number of bytes of the terminal data carried by the data report message and the number of bytes of data stored in the channel (i.e., the number of bytes of data stored in the updated channel) is calculated. If the sum of the two is greater than the second current limiting threshold, the current limiting flag of the channel is set to the second value, otherwise, there is no need to update the value of the current limiting flag.

[0114] Step S306: writing the terminal data extracted from the data reporting message into the cache.

[0115] After it is determined in step S303 that the channel in the message middleware needs to be current limited, the terminal data extracted from the data reporting message can be written into a cache, wherein the cache can be implemented in a List (queue) or other manners.

[0116] In the embodiment of the present invention, the unified collection and output of multi-terminal data is realized through the above steps, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

[0117] Figure 4 Schematic diagram of the main modules of a data processing device according to the third embodiment of the present invention. Figure 4 As shown, the data processing device 400 of the embodiment of the present invention includes: a verification module 401, a judgment module 402, and a writing module 403.

[0118] The verification module 401 is used to verify the data reporting message after receiving the data reporting message sent by the collection client.

[0119] There are multiple collection clients for collecting data from multiple terminals. For example, when it is necessary to collect data from terminal 1 (such as user behavior data on a Web page) and data from terminal 2 (such as log data from a backend server), a corresponding collection client can be deployed for terminal 1 and a corresponding collection client can be deployed for terminal 2 to collect data from multiple terminals. Afterwards, the collection client can carry the collected terminal data in a data reporting message and send it to the data processing device.

[0120] In an optional implementation, the verification module 401 verifies the data report message including: verifying the correctness of the data according to the verification code carried by the data report message. For example, the verification code carried by the data report message can be compared with a pre-stored verification code or a verification code generated in real time; if the two are consistent, it is confirmed that the data carried by the message has not been tampered with, that is, the data carried by the message is correct; if the two are inconsistent, it is confirmed that the data carried by the message has been tampered with, that is, the data carried by the message is incorrect. Through the above steps, the security of the data can be improved.

[0121] In another optional implementation, the verification module 401 verifies the data report message including: performing field integrity verification on the data report message. For example, each field in the data report message may be traversed, and if a field is missing, it is confirmed that the message has not passed the field integrity verification; if no field is missing, it is confirmed that the message has passed the field integrity verification.

[0122] In another optional implementation, the verification module 401 verifies the data report message including: verifying the user identity according to the user information carried in the data report message. For example, the user's account and password can be obtained from the data report message, and if the user's account and password match, the user is confirmed to be a legitimate user; if the user's account and password do not match, the user is confirmed to be an illegal user.

[0123] To further improve the security of data, the verification module 401 may also combine any two or three of the above three optional implementations. In addition, in specific implementation, the verification module 401 may also use other methods to verify the data reporting message.

[0124] The judgment module 402 is used to obtain corresponding communication information according to the data reporting message when the data reporting message passes the verification, and judge whether to limit the flow of the channel in the message middleware according to the channel information.

[0125] Among them, the message middleware includes multiple channels (or message pipes), and different channels can be used to store different types of data. Exemplarily, if the message middleware adopts Kafka, the multiple channels can be multiple Topics in Kafka, and different Topics can be used to store different types of data. For example, Topic1 is used to store user behavior data on Web pages, and Topic2 is used to store log data of background servers. For another example, Topic1 is used to store data collected from application 1, and Topic2 is used to store data collected from application 2.

[0126] In an optional example, the judgment module 402 may determine the channel used to store the data source data based on the data source identifier (such as the application name) carried by the data reporting message, then obtain the information of the channel, and determine whether it is currently necessary to limit the flow of the channel based on the channel information.

[0127] In an optional implementation of the above optional example, the channel information includes a current limiting flag. Among them, the current limiting flag has two possible values: a first value and a second value. For example, the first value is "no" and the second value is "yes", or the first value is "0" and the second value is "1". In this optional implementation, the judgment module 402 can determine whether it is currently necessary to limit the current of the channel by reading the value of the current limiting flag of the channel. If the current limiting flag of the channel is the first value, there is no need to limit the current of the channel; if the current limiting flag of the channel is the second value, it is necessary to limit the current of the channel.

[0128] In another optional implementation of the above optional example, the channel information includes the amount of data stored in the channel. In this optional implementation, the judgment module 402 can calculate the sum of the amount of terminal data carried by the data reporting message and the amount of data stored in the channel; if the sum of the two is greater than the preset current limiting threshold, the channel needs to be current limited; if the sum of the two is less than or equal to the preset current limiting threshold, the channel does not need to be current limited.

[0129] The writing module 403 is used to extract the collected terminal data from the data reporting message and write the terminal data into the channel without limiting the current of the channel.

[0130] For example, assuming that the channel corresponding to the data reporting request is Topic1, the terminal data extracted from the data reporting message is written into Topic1.

[0131] In the device of the embodiment of the present invention, multi-terminal data is collected based on multiple collection clients, and after receiving the data reporting message sent by the collection client, the data reporting message is verified by the verification module, and after the verification is passed, the judgment module is used to determine whether the corresponding channel needs to be limited, and when no current limiting is required, the collected terminal data is extracted from the data reporting message by the writing module and the terminal data is written into the channel, thereby realizing the unified collection and output of multi-terminal data, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

[0132] Figure 5 FIG. 4 is a schematic diagram of the main modules of a data processing system according to a fourth embodiment of the present invention. Figure 5 As shown, the data processing system 500 of the embodiment of the present invention includes: a collection client 501 , a collection client 502 , a collection client 503 , a gateway 504 , and a message middleware 505 .

[0133] Collection clients 501, 502, and 503 are deployed on multiple terminals to collect data from the multiple terminals. For different types of terminal data, data collection can be performed through different collection clients. For example, for user behavior data on a Web page, data collection can be performed through JS SDK (a data collection point tool written in JavaScript); for log data on a background server, data collection can be performed through a customized Flume.

[0134] It should be pointed out that Figure 5 The number of acquisition clients shown is exemplary and does not constitute a limitation to the present invention. In specific implementation, the number of acquisition clients can be flexibly adjusted according to actual needs.

[0135] The gateway 504 is provided with a data processing device, which is used to verify the data reporting message after receiving the data reporting message sent by any one of the multiple collection clients; it is also used to obtain the corresponding channel information according to the data reporting message when the data reporting message passes the verification, and determine whether to limit the flow of the channel in the message middleware according to the channel information; it is also used to extract the collected terminal data from the data reporting message and write the terminal data into the channel when there is no need to limit the flow of the channel.

[0136] Among them, the message middleware includes multiple channels (or message pipes), and different channels can be used to store different types of data. Exemplarily, if the message middleware adopts Kafka, the multiple channels can be multiple Topics in Kafka, and different Topics can be used to store different types of data. For example, Topic1 is used to store user behavior data on Web pages, and Topic2 is used to store log data of background servers. For another example, Topic1 is used to store data collected from application 1, and Topic2 is used to store data collected from application 2.

[0137] The message middleware 505 is used to output the terminal data written into the channel.

[0138] For example, after the entire data collection and reporting process is completed through the gateway 504, output can be enabled through Kafka for access and use by back-end data analysts.

[0139] In the embodiment of the present invention, the unified collection and output of multi-terminal data is realized through the above system, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of the data output source and improves the access efficiency of the data output source.

[0140] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0141] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0142] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0143] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0144] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a verification module, a judgment module and a writing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the verification module may also be described as a "module for verifying data reporting messages".

[0147] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes the following process: after receiving the data reporting message sent by the collection client, verify the data reporting message; wherein, there are multiple collection clients, which are used to collect data from multiple terminals; when the data reporting message passes the verification, the corresponding channel information is obtained according to the data reporting message, and it is determined whether to limit the flow of the channel in the message middleware according to the channel information; when there is no need to limit the flow of the channel, the collected terminal data is extracted from the data reporting message, and the terminal data is written into the channel.

[0148] According to the technical solution of the embodiment of the present invention, unified collection and output of multi-terminal data is realized, which not only improves the security and timeliness of multi-terminal data collection and output, but also reduces the access cost of data output source and improves the access efficiency of data output source.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: After receiving the data reporting message sent by the collection client, verifying the data reporting message; wherein there are multiple collection clients for collecting data from multiple terminals; In the case where the data report message passes the verification, the corresponding channel information is obtained according to the data report message, and it is determined whether to limit the flow of the channel in the message middleware according to the channel information; specifically comprising: determining the corresponding channel identifier according to the application identifier carried by the data report message; querying the corresponding channel information according to the channel identifier; wherein the channel information includes a flow limiting flag of the channel; if the flow limiting flag of the channel is a first value, there is no need to limit the flow of the channel in the message middleware; if the flow limiting flag of the channel is a second value, it is necessary to limit the flow of the channel in the message middleware; In the absence of current limiting on the channel, the collected terminal data is extracted from the data reporting message, and the terminal data is written into the channel.

2. The method according to claim 1, characterized in that The verifying the data reporting message includes: The correctness of the data is verified according to the verification code carried in the data reporting message, and / or the field integrity of the data reporting message is verified, and / or the user identity is verified according to the user information carried in the data reporting message.

3. The method according to claim 1, characterized in that The channel information also includes: time period, amount of data stored in the channel during the time period; The method also includes: after writing the terminal data into the channel, updating the amount of data stored in the channel within the time period according to the amount of the terminal data; when the updated amount of data stored in the channel is greater than the current limiting threshold, setting the current limiting flag of the channel to a second value.

4. The method according to claim 3, characterized in that The amount of data stored in the channel during the time period includes: the number of data items stored in the channel during the time period, and the number of bytes of data stored in the channel during the time period; The situation where the amount of data stored in the updated channel is greater than the current limiting threshold includes: the number of data items stored in the updated channel is greater than the first current limiting threshold, or the number of bytes of data stored in the updated channel is greater than the second current limiting threshold.

5. The method according to claim 1, wherein The extracting the collected terminal data from the data reporting message and writing the terminal data into the channel includes: Extracting collected terminal data from the data reporting message; reading data structure configuration information corresponding to the channel; assembling the terminal data according to the data structure configuration information, and writing the assembled data into the channel.

6. The method according to claim 1, characterized in that The method further comprises: When it is necessary to limit the flow of the channel, the data reporting message is written into the cache.

7. A data processing device, characterized in that: The device comprises: A verification module is used to verify the data reporting message after receiving the data reporting message sent by the collection client; wherein there are multiple collection clients, which are used to collect data from multiple terminals; A judgment module is used to obtain corresponding channel information according to the data reporting message when the data reporting message passes the verification, and judge whether to limit the flow of the channel in the message middleware according to the channel information; specifically used to: determine the corresponding channel identifier according to the application identifier carried by the data reporting message; query the corresponding channel information according to the channel identifier; wherein the channel information includes a flow limiting flag bit of the channel; if the flow limiting flag bit of the channel is a first value, there is no need to limit the flow of the channel in the message middleware; if the flow limiting flag bit of the channel is a second value, it is necessary to limit the flow of the channel in the message middleware; The writing module is used to extract the collected terminal data from the data reporting message and write the terminal data into the channel without limiting the current of the channel.

8. A data processing system, characterized in that: The system includes: multiple collection clients, gateways, and message middleware; The multiple collection clients are deployed on multiple terminals and are used to collect data from the multiple terminals; The gateway is provided with a data processing device, which is used to verify the data reporting message after receiving the data reporting message sent by any one of the multiple collection clients; it is also used to obtain the corresponding channel information according to the data reporting message when the data reporting message passes the verification, and determine whether to limit the flow of the channel in the message middleware according to the channel information, specifically for determining the corresponding channel identifier according to the application identifier carried by the data reporting message; querying the corresponding channel information according to the channel identifier; wherein the channel information includes the flow limiting flag of the channel; if the flow limiting flag of the channel is a first value, there is no need to limit the flow of the channel in the message middleware; if the flow limiting flag of the channel is a second value, it is necessary to limit the flow of the channel in the message middleware; it is also used to extract the collected terminal data from the data reporting message and write the terminal data into the channel when there is no need to limit the flow of the channel; The message middleware is used to output the terminal data written into the channel.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images