Big Data Processing Methods

Abstract

This invention relates to a big data processing method, comprising: acquiring real-time data from a video service system terminal, wherein the real-time data is user behavior data; classifying the real-time data into themes to generate theme behavior data packets; parsing the theme behavior data packets to generate a first field data group; determining whether the fields in the first field data group are normal; if not, sending the first field data group to an abnormal theme behavior data packet set; if yes, acquiring business data from the video service system server; associating the first field data group and the business data to generate a second field data group; and sending the second field data group to a data warehouse for storage pending application interface calls. This invention's big data processing method performs real-time streaming processing on real-time reported data, transforming informal data into formal data that can be called by application interfaces in real time, thereby saving computing and storage resources, accelerating the generation of operational strategies, and improving operational efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a big data processing method. Background Art

[0002] Interactive Internet Protocol Television (IPTV) is a new media service that transmits content including interactive Internet television, graphics and data over a broadband network, providing home users with a variety of interactive services including digital television.

[0003] At present, the IPTV system is divided into two modules, offline and real-time, for big data processing. There are a large number of repeated processing links in the two modules, resulting in a waste of computing and storage resources. The data processing of the real-time module is mainly aimed at the output of single indicators such as the data screen, while indicators such as data reports are realized through the offline module. The offline module adopts the T+1 offline processing mode, and reports and various indicators are output at regular intervals. Due to the delay in reporting data, it often leads to a lag in operation strategy and unsatisfactory operation results.

[0004] In addition to the IPTV system, other video service systems, such as the big data processing of the OTT (Over-The-Top) system, will also have the same problem. How to solve the above problem is the focus of the present invention. Summary of the invention

[0005] The purpose of the present invention is to provide a big data processing method to address the defects of the prior art, to perform real-time streaming processing on real-time reported data, and to convert informal data into formal data that can be called by application interfaces in real time, thereby saving computing and storage resources, accelerating the generation of operation strategies, and improving operation results.

[0006] To achieve the above object, the present invention provides a big data processing method, the method comprising:

[0007] Acquire real-time data of the video service system terminal, wherein the real-time data is user behavior data;

[0008] Classifying the real-time data by subject and generating subject behavior data packets;

[0009] Parsing the subject behavior data packet to generate a first field data group;

[0010] Determining whether the fields in the first field data group are normal;

[0011] If not, sending the first field data group to the abnormal subject behavior data packet set;

[0012] If yes, obtain the service data of the video service system server;

[0013] Associating the first field data group with business data and generating a second field data group;

[0014] The second field data group is sent to a data warehouse for storage and waits for application interface call.

[0015] Preferably, before parsing the subject behavior data packet to generate the first field data group, the method further includes:

[0016] Determining whether the subject behavior data packet is in a preset data format;

[0017] If yes, parse the subject behavior data packet to generate a first field data group;

[0018] If not, the subject behavior data packet is sent to the abnormal subject behavior data packet set.

[0019] Preferably, the types of the subject behavior data packets include: page browsing subject behavior data packets, recommended position click subject behavior data packets, power-on subject behavior data packets, user online heartbeat subject behavior data packets, live broadcast subject behavior data packets, on-demand subject behavior data packets, and replay subject behavior data packets.

[0020] Preferably, the parsing of the subject behavior data packet to generate the first field data group specifically includes: performing database field parsing on the subject behavior data packet to generate the first field data group.

[0021] Preferably, the determining whether the fields in the first field data group are normal is specifically: determining whether the key fields in the first field data group are empty, have abnormal values ​​or are garbled.

[0022] Preferably, whether the first field data group needs to be associated with the business data is determined according to the type of the subject behavior data packet of the first field data group;

[0023] Preferably, associating the first field data group with the business data and generating the second field data group is specifically performed by taking a union of fields in the first field data group and fields in the business data to generate the second field data group.

[0024] Preferably, the sending of the second field data group to the data warehouse for storage and waiting for application interface call is specifically: the second field data group is stored in the data warehouse according to the partition of the data acquisition date.

[0025] Preferably, the sending of the second field data group to a data warehouse for storage and waiting for an application interface call is specifically: sending a plurality of the second field data groups to data warehouses of a plurality of servers for distributed storage.

[0026] The big data processing method provided by the embodiment of the present invention performs real-time streaming processing on real-time reported data, converting informal data into formal data that can be called by application interfaces in real time, thereby saving computing and storage resources, accelerating the generation of operation strategies, and improving operation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 One of the flow charts of a big data processing method provided by an embodiment of the present invention;

[0028] Figure 2 The present invention provides a second flowchart of a method for processing big data in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0030] The video service system may include an IPTV system, an OTT system, or other network-based smart TV systems. The method of the present invention may be applied to each of the above-mentioned video service systems. For ease of description, the IPTV system is used as a specific example for illustration. It is understood that the method may also be used in OTT or other network-based smart TV systems.

[0031] Figure 1 One of the flow charts of a big data processing method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0032] Step 110, obtaining real-time data of the video service system terminal, where the real-time data is user behavior data.

[0033] Specifically, the video service system terminal may include an IPTV terminal, an OTT terminal or other network-based smart TV terminal. The present invention uses an IPTV terminal as a specific example for explanation. When the IPTV system is in normal operation, the user operates the user control device or watches the video service system, and the user's behavior data will be reported through the IPTV terminal. The reported behaviors include: user power-on behavior, browsing the Electronic Program Guide (EPG) page behavior, on-demand behavior, live broadcast behavior, playback behavior, user online heartbeat behavior, etc. The reported behavior data is transmitted in a pre-set data format, and the user's online heartbeat behavior is reported by the heartbeat probe in a preset reporting time period, such as reporting every 5 minutes or 10 minutes. The interval is specifically set according to the real-time requirements of the system data and is not specifically limited. Optionally, the reported behavior data is reported in JSON format, and the reported behavior data fields are pre-set according to the different reported behaviors. In a specific example, when a user turns on the device, the reported content fields include: user IPTV account, set-top box serial number, set-top box model, city code, user IP, template name, user group, user capability platform, user operator, probe version number, EPG mode, object type, and original city code value. Preferably, the user's behavior data can be collected through the Flume system.

[0034] Step 120, classify the real-time data by subject, and generate subject behavior data packets.

[0035] Specifically, the IPTV system is aimed at multiple massive users, so there is a massive amount of real-time data reported at the same time. Each reported data of the same topic is placed in the same message queue, and the massive data is classified by topic and processed in a distributed manner. The classified data can be directly provided to multiple data consumers in the form of a unified interface service.

[0036] Preferably, the types of subject behavior data packets include: page browsing subject behavior data packet, recommendation position click subject behavior data packet, power-on subject behavior data packet, user online heartbeat subject behavior data packet, live broadcast subject behavior data packet, on-demand subject behavior data packet, and playback subject behavior data packet.

[0037] Step 130: parse the subject behavior data packet to generate a first field data group.

[0038] Specifically, the subject behavior data packet is parsed for the fields of the database to generate a first field data group. The subject behavior data packet is classified according to a data format preset when acquiring behavior data, such as JSON format. When the subject behavior data packet is sent in JSON format, the subject behavior data packet in JSON format is parsed and mapped into data of the corresponding database table field one by one to generate a first field data group.

[0039] In a preferred example, Figure 2 As shown, before step 130, the following steps are also included:

[0040] Step 210, determining whether the subject behavior data packet is in a preset data format.

[0041] If yes, proceed to step 130 .

[0042] Step 220: If not, send the subject behavior data packet to the abnormal subject behavior data packet set.

[0043] Specifically, the preset data format facilitates the transmission of collected real-time data. Preferably, the data format is defined as JSON, which has a simple data structure and is easy to parse. When a transmitted subject behavior data packet is not in the preset data format, the data packet is abandoned and sent to the abnormal subject behavior data packet set. When the same type of error data packets reaches the preset threshold, the error data packet log can be analyzed to solve the system problem.

[0044] Step 140: determine whether the fields in the first field data group are normal.

[0045] Step 150: If not, send the first field data group to the abnormal subject behavior data packet set.

[0046] Specifically, determine whether the key fields in the first field data group are empty, the uploaded values ​​are abnormal, or garbled. Since a reported real-time data contains multiple fields, the key fields in the first field data group can be defined according to the needs of different businesses of the final application interface. Each behavior data has basic fields, such as operators, user accounts, regions, and versions. They are not considered, and only the data in the key fields is judged. The multiple key fields in the first field data group need to meet the requirements of no abnormalities at the same time, which is normal, otherwise it is abnormal. Each reported data is cleaned by judging the key fields to avoid the next step of processing invalid data. The first field data group judged to be invalid is sent to the abnormal subject behavior data packet set for logging, which can be used for query and further troubleshooting.

[0047] Step 160: If yes, obtain the service data of the video service system server.

[0048] Step 170: associate the first field data group with the business data and generate a second field data group.

[0049] Specifically, when the key field in the first field data group is normal, the business data of the video service system server is obtained. In the IPTV business, the upper-level platform sends the business data to the database of each provincial platform in the form of a data stream. The business data includes media resource work order theme data, user information theme data, subscription theme data, etc. When the video service system collects the business data from the provincial platform database for the first time, it collects the full amount and puts it into the corresponding field of the database. Optionally, the corresponding fields include: operator, user account, channel code, region, etc. When the business data has not changed, the system only collects the behavior data in real time, and monitors the business data in the provincial platform database in real time to see if there is any change. If there is a change, the changed part will be collected and put into the corresponding field of the database.

[0050] In a preferred example, before step 170, the method further includes:

[0051] Whether it is necessary to associate the first field data group with the business data is determined according to the type of the subject behavior data packet of the first field data group.

[0052] Specifically, in the first field data group parsed from the subject behavior data packet, there is a type of field in the first field data group that can be directly called by the application interface, for example: the type of the power-on subject behavior data packet is power-on, and the field attributes corresponding to the power-on type data do not require additional business data, so the power-on subject behavior data packet can be directly entered into the data warehouse after parsing and waiting for calling. There is a type of field that needs to be further associated with business data and then provided to the application interface call, for example: the type of the on-demand subject behavior data packet is on-demand, and the on-demand type data needs to be associated with business data for application interface calls. In a specific example, the fields in the business data include: channel name, channel code, channel group, channel type; the fields of the first field data group after the live subject behavior data packet is parsed include: operator, user account, channel code and region. When the application port needs to obtain the number of plays and the number of people playing a live channel on that day, the fields of the first field data group cannot meet the business needs. There is no channel name in the fields of the first field data group. When displaying data, the channel code cannot be used for display, and the Chinese name must be used. Therefore, it is necessary to associate the fields of the business data with the first field data group to generate the second field data group. In a preferred example, the fields in the first field data group and the fields in the business data are taken as the union to generate a second field data group. When the fields of the first field data group after parsing the live theme behavior data packet in the above specific example are taken as the union with the fields of the business data, the fields after generating the second field data group include: channel name, channel code, channel group, channel type, operator, user account, channel code and region. After the behavior data after association has the fields of business data, when the application interface calculates the number of on-demand behaviors of a channel within 24 hours and displays it, the Chinese name of the channel can be obtained through the channel name field of the second field data group, and the Chinese name is displayed on the display terminal.

[0053] Step 180: Send the second field data group to the data warehouse for storage and wait for application interface call.

[0054] Specifically, the application interfaces include: data screen interface, data analysis interface, user portrait interface, recommendation system interface, and external interface services.

[0055] In yet another preferred example, the second field data group is stored in the data warehouse partitioned according to the data acquisition date.

[0056] Specifically, the data warehouse is partitioned into different directories by time to store the second field data group. The data in each partition is isolated from each other. When the application interface needs to call the data group of a certain day, the data warehouse will not scan the data of other days. The target data group can be found quickly to improve the working efficiency of the application interface.

[0057] In yet another preferred example, multiple second field data groups are sent to data warehouses of multiple servers for distributed storage.

[0058] Specifically, when multiple second field data groups are continuously generated and stored, distributed storage is performed according to preset rules, and optionally, distributed storage is performed according to time periods or in a polling manner, etc. In a specific example of applying distributed storage in a polling manner, it is assumed that there are three distributed servers for storing second field data groups, namely server 1, server 2, and server 3, and there are six second field data groups A, B, C, D, E, and F to be placed in the storage warehouse of the server in sequence, then A, B, and C are placed in server 1, server 2, and server 3 respectively, and D, E, and F are placed in server 1, server 2, and server 3 respectively, and so on, so that multiple second field data groups are distributedly stored. When the application interface needs to perform data calculations, it can use the data warehouse resources of each node server for distributed computing to improve computing efficiency. At the same time, when the interface calls the second field data group for calculation, it needs to use the memory and CPU resources of other servers. If the data is placed on one server, when performing calculations, it is necessary to call the computing resources on the storage data server to other servers for calculation, which will increase the data transmission volume and the calculation time, and easily cause blockage of the data reading and writing links. The use of distributed storage can solve the above problems.

[0059] In a specific example, after step 180, the application system calls the second field data group through the application interface including:

[0060] Determine the input parameters and return result set of the application interface.

[0061] Specifically, the interface services provided by the application interface include: data screen interface, data analysis interface, user portrait interface, recommendation system interface, and external interface service. The application system includes: data screen module, real-time monitoring module, behavior analysis module, audience analysis module, and statistical report module, among which the data screen module includes: overview unit, heat map unit, program real-time data unit, and channel real-time data unit; the real-time monitoring module includes: real-time user data unit, real-time audience data unit, real-time subscription data unit, and regional data indicator unit; the behavior analysis module includes: page analysis unit, recommendation unit, and search hot word unit; the audience analysis module includes: channel live broadcast unit, channel playback unit, and program on-demand unit; the statistical report module includes: trigger subscription report unit, page access report unit, recommendation unit, user active data report unit, channel live broadcast report unit, channel playback report unit, and program on-demand report unit. In a specific example, when the audience analysis module needs to call data, the interface services that need to be called include the data analysis interface, the user portrait interface, and the recommendation system interface. The interface addresses are written into the interface service of the video analysis module, and the fields of the return result set are generated based on the data required by the channel live unit, the channel playback unit, and the program on demand unit.

[0062] Generate interface instructions based on input parameters and returned result sets.

[0063] Specific interface instructions include accessing the interface address and reading the fields of the returned result set.

[0064] Get parameters and return result values ​​according to interface instructions.

[0065] Specifically, when the application module needs to call related data for calculation, it accesses the related interface at the specified address according to the interface instruction to obtain the field value set in the returned result set.

[0066] The big data processing method provided by the embodiment of the present invention performs streaming processing on the real-time acquisition, classification, analysis, rule-based cleaning, association and warehousing of the real-time data of video users, and reduces data transmission and improves computing efficiency through distributed batch streaming processing of data; the stored data is extracted through a unified interface service, which reduces the difficulty of data extraction and improves the development efficiency of the application end.

[0067] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0068] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0069] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing big data, characterized in that: The big data processing method comprises: Acquire real-time data of the video service system terminal, wherein the real-time data is user behavior data; Classifying the real-time data by subject and generating subject behavior data packets; Parsing the subject behavior data packet to generate a first field data group; Determining whether the fields in the first field data group are normal; If not, sending the first field data group to the abnormal subject behavior data packet set; If yes, obtain the service data of the video service system server; Associating the first field data group with business data and generating a second field data group; Sending the second field data group to a data warehouse for storage and waiting for application interface call; The real-time data is classified by subject to generate subject behavior data packets, specifically: Put the data reported on the same topic into the same message queue, classify the massive data reported on the same time into topics and process them in a distributed manner, and directly provide the classified data to multiple data consumers in the form of a unified interface service. Before associating the first field data group with the business data and generating the second field data group, the method further includes: Determine whether it is necessary to associate the first field data group with business data according to the type of the subject behavior data packet of the first field data group; if the type of the subject behavior data packet does not require additional business data, directly put the subject behavior data packet into the data warehouse after parsing and waiting for calling; The first field data group is associated with the business data, and a second field data group is generated, specifically: If the type of the subject behavior data packet needs to be further associated with the fields of the business data, the fields of the first field data group after the live subject behavior data packet is parsed are associated with the fields of the business data to generate a second field data group.

2. The big data processing method according to claim 1, characterized in that: Before parsing the subject behavior data packet to generate the first field data group, the method further includes: Determining whether the subject behavior data packet is in a preset data format; If yes, parse the subject behavior data packet to generate a first field data group; If not, the subject behavior data packet is sent to the abnormal subject behavior data packet set.

3. The big data processing method according to claim 1, characterized in that: The types of the subject behavior data packets include: page browsing subject behavior data packets, recommendation position click subject behavior data packets, power-on subject behavior data packets, user online heartbeat subject behavior data packets, live broadcast subject behavior data packets, on-demand subject behavior data packets, and playback subject behavior data packets.

4. The big data processing method according to claim 1, characterized in that: The subject behavior data packet is parsed to generate a first field data group, specifically: The subject behavior data packet is subjected to a field analysis of a database to generate a first field data group.

5. The big data processing method according to claim 1, characterized in that: The determining whether the fields in the first field data group are normal is specifically: Determine whether the key field in the first field data group is empty, has an abnormal value, or is garbled.

6. The big data processing method according to claim 1, characterized in that: The first field data group is associated with the business data, and a second field data group is generated, specifically: A union of the fields in the first field data group and the fields in the business data is taken to generate a second field data group.

7. The big data processing method according to claim 1, characterized in that: The sending of the second field data group to the data warehouse for storage and waiting for application interface call is specifically: The second field data group is stored in the data warehouse according to partitions of data acquisition dates.

8. The big data processing method according to claim 1 or 7, characterized in that: The sending of the second field data group to the data warehouse for storage and waiting for application interface call is specifically: Sending a plurality of the second field data groups to data warehouses of a plurality of servers for distributed storage.

Images