Method and apparatus for data processing

Abstract

The present invention discloses a method and apparatus for data processing, relating to the field of computer technology. A specific embodiment of the method includes: obtaining object data of a session window, where the object data includes: a session identifier and an object identifier; determining whether the object identifier exists in a window object set corresponding to the session identifier; if it exists, no processing is performed on the object data; otherwise, processing is performed on the object data. This embodiment can achieve computing processing only when there are new elements in the session window by performing processing on the object data only when the object identifier of the object data does not exist in the window object set, greatly reducing the amount of data processing and facilitating the user to view real-time data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method and apparatus for data processing. Background Art

[0002] Currently, most network platforms only provide index queries for offline data. Business parties have a strong demand for real-time data. With only offline data and no real-time data, the timeliness of data viewing is poor, and the demands of business parties for viewing real-time data cannot be met. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for data processing. By processing the object data only when the object identifier of the object data does not exist in the window object set, it is possible to perform calculation processing only when there are new elements in the session window, greatly reducing the amount of data processing and facilitating users to view real-time data.

[0004] To achieve the above object, according to one aspect of the embodiments of the present invention, a method for data processing is provided, including:

[0005] Obtain object data of a session window, where the object data includes: a session identifier and an object identifier; determine whether the object identifier exists in a window object set corresponding to the session identifier; if it exists, do not process the object data; otherwise, process the object data.

[0006] Optionally, the window object set is stored in the form of a HashMap, the key of the HashMap is the session identifier, and the value of the HashMap is the object identifiers of each window object included in the session window.

[0007] Optionally, the object identifier is the hash value of the object data.

[0008] Optionally, before obtaining the object data of the session window, it further includes: using Kafka to save the object data of the collected session window;

[0009] Obtaining the object data of the session window includes: periodically consuming the messages in the Kafka to obtain the object data of the session window.

[0010] Optionally, before using Kafka to save the object data of the collected session window, it further includes: cleaning redundant fields in the object data, and / or, unifying the field names in the object data.

[0011] Optionally, after processing the object data, it further includes: writing the processed result data into Doris.

[0012] Optionally, write the processed result data to Doris, including:

[0013] Confirm that the data volume of the result data is greater than the set data volume threshold, and / or confirm that the generation time of the result data is greater than the set time threshold.

[0014] According to another aspect of the embodiments of the present invention, there is provided a data processing device, including:

[0015] An acquisition module, which acquires object data of a session window, and the object data includes: a session identifier and an object identifier;

[0016] A judgment module, which judges whether the object identifier exists in the window object set corresponding to the session identifier;

[0017] A processing module, when the object identifier exists in the window object set corresponding to the session identifier, does not process the object data; when the object identifier does not exist in the window object set corresponding to the session identifier, processes the object data.

[0018] Optionally, the window object set is stored in the form of a HashMap, the key of the HashMap is the session identifier, and the value of the HashMap is the object identifier of each window object included in the session window.

[0019] Optionally, the object identifier is the hash value of the object data.

[0020] Optionally, the acquisition module is further configured to: before acquiring the object data of the session window, use Kafka to save the acquired object data of the session window;

[0021] Acquiring the object data of the session window includes: periodically consuming the messages in the Kafka to obtain the object data of the session window.

[0022] Optionally, the acquisition module is further configured to: before using Kafka to save the acquired object data of the session window, clean the redundant fields in the object data, and / or unify the field names in the object data.

[0023] Optionally, the processing module is further configured to: after processing the object data, write the processed result data to Doris.

[0024] Optionally, the processing module writes the result data after the processing to Doris, including: confirming that the data volume of the result data is greater than a set data volume threshold, and / or confirming that the generation time of the result data is greater than a set time threshold.

[0025] According to another aspect of the embodiments of the present invention, there is provided an electronic device for data processing, including:

[0026] One or more processors;

[0027] A storage device for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by the present invention.

[0029] According to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the data processing method provided by the present invention is implemented.

[0030] One embodiment of the above invention has the following advantages or beneficial effects: By performing processing on the object data only when the object identifier of the object data does not exist in the window object set, it is possible to achieve calculation processing only when there are new elements in the session window, greatly reducing the data processing volume and facilitating users to view real-time data.

[0031] The further effects of the above non-conventional optional methods will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Among them:

[0033] Figure 1 is an exemplary system architecture diagram suitable for the data processing method or data processing device of the embodiments of the present invention;

[0034] Figure 2 is a schematic diagram of the main process of the data processing method of the embodiments of the present invention;

[0035] Figure 3 is a schematic diagram of the data processing architecture in an optional embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of the data processing process in an optional embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of the main modules of the data processing device of the embodiments of the present invention;

[0038] Figure 6 It is a schematic structural diagram of a computer system of a terminal device or a server suitable for implementing the embodiments of the present invention. Detailed implementation manners

[0039] The following describes exemplary embodiments of the present invention with reference to the accompanying drawings. Various details of the embodiments of the present invention are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, descriptions of well-known functions and structures are omitted for clarity and conciseness.

[0040] Figure 1 An exemplary system architecture diagram of a method for data processing or a device for data processing suitable for application to the embodiments of the present invention is shown. As Figure 1 shown, the exemplary system architecture of the method for data processing or the device for data processing in the embodiments of the present invention includes:

[0041] As Figure 1 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0042] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0043] The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0044] The server 105 may be a server providing various services, such as a background management server (for example only) that supports shopping websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process data such as product information query requests received, and feedback the processing results (such as target push information, product information - for example only) to the terminal devices 101, 102, 103.

[0045] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by the server 105. Correspondingly, the data processing device is generally disposed in the server 105.

[0046] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0047] Figure 2 are merely illustrative. According to actual needs, there can be any number of terminal devices, networks, and servers. Figure 2 is a schematic diagram of the main process of the data processing method according to the embodiments of the present invention. As

[0048] shown, the data processing method includes step S201, step S202, and step S203.

[0049] A session refers to the process of a terminal user communicating with an interactive system. For example, the process from entering the operating system by inputting an account password to exiting the operating system is a session process. The session identifier is used to uniquely represent a session, such as the UUID (Universally Unique Identifier) of the session. The window duration of each session can be selectively set according to actual situations, such as 10 minutes or 30 minutes.

[0050] A session often involves multiple data objects, and the data related to the object is called object data. The object identifier is used to uniquely represent a piece of object data. Optionally, the object identifier is the hash value of the object data.

[0051] Step S202, determine whether the object identifier exists in the window object set corresponding to the session identifier. If it exists, jump to step S203 and do not process the object data; otherwise, jump to step S204 and process the object data.

[0052] The processing mentioned here refers to obtaining result data by processing the collected data. The calculation logic of the processing can be selectively set according to actual situations. For example, calculate the following indicators: page access stay duration, whether the session is the first page request flag, whether the session is the last page request flag, and the access depth of the session.

[0053] The present invention can achieve calculation processing only when there are new elements in the session window by processing the object data only when the object identifier of the object data does not exist in the window object set, greatly reducing the data processing volume and facilitating users to view real-time data.

[0054] Optionally, the set of window objects is stored in the form of a HashMap (a key-value based data structure). The key of the HashMap is the session identifier, and the value of the HashMap is the object identifier of each window object included in the session window. Storing in the form of a HashMap facilitates querying and has stable performance.

[0055] Optionally, before obtaining the object data of the session window, it further includes: using Kafka (a distributed publish-subscribe messaging system that can process the action stream data of users on the website) to save the object data of the collected session window. Using Kafka to save the object data of the collected session window can achieve distributed execution of data collection and processing. Obtaining the object data of the session window may include: periodically consuming the messages in the Kafka to obtain the object data of the session window. For example, in order to improve the timeliness of data, a trigger can be triggered every two minutes for calculation.

[0056] Before using Kafka to save the object data of the collected session window, it may further include: cleaning the redundant fields in the object data, and / or, unifying the field names in the object data. By cleaning the redundant fields in the object data, the dirty data in the object data can be removed, facilitating subsequent processing. By unifying the field names in the object data, it also facilitates subsequent processing.

[0057] After processing the object data, it may further include: writing the processed result data into Doris (an online analytical processing data storage and computing engine). In order to ensure that the data in the entire process is consumed only once, inherit the GenericWriteAheadSink class. First, store the checkpointId in Redis. When the write to Doris is successful, then complete the entire checkpoint. At the same time, the model table of Doris can store data using the unique key model. Since the stored wide table detail data is stored, it can support various dimension combination aggregation queries, facilitating data analysis from various perspectives.

[0058] Optionally, writing the processed result data into Doris includes: confirming that the data volume of the result data is greater than the set data volume threshold, and / or, confirming that the generation time of the result data is greater than the set time threshold. For example, when the data exceeds 80M or the interval time exceeds 1 minute, it is written into Doris. By setting the quantity threshold and time threshold, the number of writes to Doris can be reduced, reducing the impact on the performance of Doris. In order to prevent a machine failure or slowness, the method of polling the machine IP can be used for retry to facilitate timely discovery and positioning.

[0059] The data processing method in the embodiments of the present invention can provide users with real-time data, facilitating users to view the real-time data status of traffic in real time and providing support for users to make quick decisions on operations.

[0060] Figure 3 It is a schematic diagram of the data processing architecture in an alternative embodiment of the present invention. As Figure 3 shown. ClickStream refers to the trajectory of a user's continuous access on a website. In this embodiment, log data is first collected according to the click stream, and then the log data is preprocessed by a sending proxy, including cleaning redundant fields, unifying field names, etc., and outputting unified and standardized detailed data and sending it to Kafka. Flink parses the detailed data of Kafka, processes the data into a wide table and stores it in Doris, and based on the Doris engine, a system page is developed for real-time data display. Figure 4 It is a schematic diagram of the data processing process in an alternative embodiment of the present invention. In this embodiment, Flink session window is used to process object data, and grouping is performed according to the session identifier during processing ( Figure 4 group by sessionid in Figure 4 ). Processing is performed every N seconds ( Figure 4 trigger computerevery N seconds in

[0061] or when the session ends ( Figure 4 trigger computer when windows end in

[0061] ). The calculation metrics for processing include: page access stay duration, flag indicating whether this session is the first page request, flag indicating whether this session is the last page request, and access depth of this session. Custom calculation logic is designed to perform processing calculations only when new elements are generated in the window. Through this optimization, the number of calculations is reduced by more than 80%. The specific implementation is as follows: Design a HashMap with a fixed size (MaxSize) to store objects in the window, where the key is the uuid of the session, and the value is a HashSet. The HashSet stores the hashCode values of MaxSize elements in the session window. When calculating the window, first query whether the HashSet with the key uuid in the HashMap contains the hashCode of the element. If it contains, it is not a new element and does not participate in this window calculation; if it does not contain, it is a new element and participates in this processing calculation.

[0061] The custom implementation of the sink (an operation method that can be used to process calculation results, such as console output or saving to a database) to write data into Doris can ensure the Exactly Once semantics. To ensure that the data in the entire process is consumed only once, inherit from the GenericWriteAheadSink class, first store the checkpointId in Redis (Remote Dictionary Server, which is a Key-Value database), and when the write to Doris is successful, complete the entire checkpoint. At the same time, the model table in Doris uses the unique key model ( Figure 4 the UNIQUE KEY data model in

[0062] Figure 4 to store data. The Checkpoint in

[0063] is an internal event. After this event is activated, it will trigger the database writer process (DBWR) to write the dirty data blocks in the data buffer (DATABUFFER CACHE) to the data file.

[0064] Figure 5 is a schematic diagram of the main modules of the data processing device according to an embodiment of the present invention. As Figure 5 shown, the data processing device 500 includes:

[0065] An acquisition module 501 that acquires the object data of the session window, and the object data includes: a session identifier and an object identifier;

[0066] A judgment module 502 that judges whether the object identifier exists in the window object set corresponding to the session identifier;

[0067] A processing module 503 that, when the object identifier exists in the window object set corresponding to the session identifier, does not process the object data; when the object identifier does not exist in the window object set corresponding to the session identifier, processes the object data.

[0068] Optionally, the window object set is stored in the form of a HashMap, the key of the HashMap is the session identifier, and the value of the HashMap is the object identifiers of the respective window objects included in the session window.

[0069] Optionally, the object identifier is the hash value of the object data.

[0070] Optionally, the obtaining module is further configured to: before obtaining the object data of the session window, save the collected object data of the session window using Kafka;

[0071] Obtaining the object data of the session window includes: periodically consuming the messages in the Kafka to obtain the object data of the session window.

[0072] Optionally, the obtaining module is further configured to: before saving the collected object data of the session window using Kafka, clean the redundant fields in the object data and / or unify the field names in the object data.

[0073] Optionally, the processing module is further configured to: after processing the object data, write the processed result data into Doris.

[0074] Optionally, the processing module writes the processed result data into Doris, including: confirming that the data volume of the result data is greater than a set data volume threshold and / or confirming that the generation time of the result data is greater than a set time threshold.

[0075] Figure 6 It is a schematic structural diagram of a computer system of a terminal device suitable for implementing the embodiments of the present invention. As Figure 6 shown, the computer system 600 of the terminal device in the embodiments of the present invention includes:

[0076] It includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage section 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0077] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. The drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that the computer program read from it can be installed into the storage section 608 as needed.

[0078] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 609 and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are performed.

[0079] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0081] The modules described in the embodiments of the present invention can be implemented in software or in hardware. The described modules can also be provided in a processor. For example, it can be described as: a processor includes an acquisition module, a judgment module, and a processing module. Among them, the names of these modules do not constitute a limitation on the module itself in some cases. For example, the acquisition module can also be described as "a module for processing the object data".

[0082] On the other hand, the present invention also provides a computer-readable medium, which can be included in the device described in the above embodiments; or can exist separately without being assembled into the device. The above computer-readable medium carries one or more programs. When the one or more programs are executed by the device, the device includes: acquiring object data of a session window, where the object data includes: a session identifier and an object identifier; judging whether the object identifier exists in a window object set corresponding to the session identifier; if it exists, not processing the object data; otherwise, processing the object data.

[0083] According to the technical solution of the embodiments of the present invention, by processing the object data only when the object identifier of the object data does not exist in the window object set, it is possible to achieve that calculation processing is performed only when there are new elements in the session window, greatly reducing the data processing volume and facilitating the user to view real-time data.

[0084] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that, it includes: Obtaining object data of a session window every N seconds or at the end of a session, where the object data includes: a session identifier and an object identifier; a session refers to the process of a terminal user communicating with an interactive system; Determining whether the object identifier exists in the window object set corresponding to the session identifier; the window object set stores the object identifiers of a fixed number of window objects; If it exists, no processing is performed on the object data; otherwise, the object data is processed.

2. The method according to claim 1, characterized in that, The window object set is stored in the form of a HashMap, the key of the HashMap is the session identifier, and the value of the HashMap is the object identifier of each window object included in the session window.

3. The method according to claim 2, characterized in that, The object identifier is the hash value of the object data.

4. The method according to claim 1, characterized in that, Before obtaining the object data of the session window, it further includes: using Kafka to save the object data of the collected session window; Obtaining the object data of the session window includes: periodically consuming the messages in the Kafka to obtain the object data of the session window.

5. The method according to claim 4, characterized in that, Before using Kafka to save the object data of the collected session window, it further includes: cleaning redundant fields in the object data, and / or, unifying the field names in the object data.

6. The method according to claim 1, characterized in that, After processing the object data, it further includes: writing the processed result data into Doris.

7. The method according to claim 6, characterized in that, Writing the processed result data into Doris includes: Confirming that the data volume of the result data is greater than a set data volume threshold, and / or, confirming that the generation time of the result data is greater than a set time threshold.

8. A data processing device, characterized in that, it includes: An obtaining module that obtains object data of a session window every N seconds or at the end of a session, where the object data includes: a session identifier and an object identifier; a session refers to the process of a terminal user communicating with an interactive system; A judging module that judges whether the object identifier exists in the window object set corresponding to the session identifier; the window object set stores the object identifiers of a fixed number of window objects; A processing module that does not process the object data when the object identifier exists in the window object set corresponding to the session identifier; and processes the object data when the object identifier does not exist in the window object set corresponding to the session identifier.

9. A data processing electronic device, characterized in that, it includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, when the program is executed by a processor, the method according to any one of claims 1-7 is implemented.

Images