A method and device for data processing

Abstract

The present invention discloses a method and device for data processing, and relates to the field of computer technology. A specific implementation of the method includes obtaining page configuration content, parsing it into a script string based on the page configuration rules, and then loading the script string through a preset custom function; reading source data and inputting it into a script, executing the script through the custom function, determining whether the source data meets the preset conditions, and performing screening of the source data. Therefore, the implementation of the present invention can solve the problem of poor timeliness of existing mass data filtering.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] Currently, most of the filtering of massive data is based on Hadoop to implement data processing and filtering under specific rules by writing code, such as writing MapReduce programs, HIVE scripts, etc.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] When filtering massive amounts of data using rules, if the rules change, the code needs to be changed to re-implement the rules. Therefore, when there are many rules and they change frequently, the timeliness requirement cannot be met at all. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for data processing, which can solve the problem of poor timeliness in existing mass data filtering.

[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for data processing is provided, including obtaining page configuration content, parsing it into a script string based on the page configuration rules, and then loading the script string through a preset custom function; reading source data and inputting it into a script, executing the script through the custom function, determining whether the source data meets preset conditions, so as to perform screening of the source data.

[0007] Optionally, parse into a script string based on page configuration rules, including:

[0008] The page is read through JavaScript, concatenated into a specific expression string, and the expression string is stored in the HIVE table.

[0009] Optionally, before loading the script string through a preset custom function, the following steps are included:

[0010] Write UDF functions in Hadoop to generate custom functions.

[0011] Optionally, the script string is loaded through a preset custom function, including:

[0012] Read the HIVE table through the preset custom function, extract the script string, and then load the script string through Groovy.

[0013] Optionally, read source data into the script, including:

[0014] According to the variable name in the script string, find the value consistent with the variable name in the field name array, and record the position of the value in the field name array;

[0015] Replace the variable in the script string with the value at the same position in the field value array.

[0016] Optionally, executing the script through the custom function to determine whether the source data meets a preset condition includes:

[0017] Start Groovy to execute the script through the custom function, obtain the execution result and store it in the corresponding position of the source data in the HIVE table;

[0018] Based on the execution result, determine whether the source data meets the preset conditions.

[0019] Optionally, read source data before inputting it into the script, including:

[0020] Write HIVE scripts to query source data in HDFS.

[0021] In addition, the present invention also provides a data processing device, including an acquisition module, used to obtain page configuration content, parse it into a script string based on the page configuration rules, and then load the script string through a preset custom function; a processing module, used to read source data and input it into a script, execute the script through the custom function, determine whether the source data meets the preset conditions, so as to perform screening of the source data.

[0022] An embodiment of the above invention has the following advantages or beneficial effects: because the page configuration content is obtained, it is parsed into a script string based on the page configuration rules, and then the script string is loaded through a preset custom function; the source data is read and input into the script, and the script is executed through the custom function to determine whether the source data meets the preset conditions, so as to perform technical means for screening the source data. Therefore, the technical problem of poor timeliness of existing massive data filtering is overcome, and the technical effect of not needing to develop support rules is achieved, which can not only make the rules effective in real time, but also save labor costs.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0025] Figure 1 is a schematic diagram of the main flow of the data processing method according to the first embodiment of the present invention;

[0026] Figure 2 is a framework diagram of a method for data processing according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of the main flow of a data processing method according to another embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0029] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0030] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] Figure 1 is a schematic diagram of the main flow of the data processing method according to the first embodiment of the present invention, such as Figure 1 As shown, the data processing method includes:

[0033] Step S101, obtaining page configuration content, parsing it into a script string based on page configuration rules, and then loading the script string through a preset custom function.

[0034] In some embodiments, based on the page configuration rules (for example, a WEB page is configured with arithmetic expressions and logical expressions), the page configuration content is read through JavaScript, spliced ​​into a specific expression string, and the expression string is stored in the HIVE table. For example: "age>10and level>20".

[0035] Among them, JavaScript (referred to as "JS") is a lightweight, interpreted or just-in-time compiled programming language with function priority. HIVE is a data warehouse tool based on Hadoop, which can map structured data files into a database table and provide simple SQL query functions, which can convert SQL statements into MapReduce tasks for execution.

[0036] It is worth noting that a UDF function is written in Hadoop to generate a custom function, and then the script string is loaded through a preset custom function. Among them, Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. The full name of the UDF function in HIVE is User Defined Function, that is, a user-defined function.

[0037] In addition, the HIVE table can be read through a preset custom function, the script string can be taken out, and then the script string can be loaded through Groovy.

[0038] Step S102, reading source data and inputting it into a script, executing the script through the user-defined function, determining whether the source data meets a preset condition, and performing screening on the source data.

[0039] In some embodiments, the source data is read and input into the script, and the specific implementation process includes:

[0040] According to the variable name in the script string, find the value that matches the variable name in the field name array, and record the position of the value in the field name array. Take the value at the same position in the field value array to replace the variable in the script string.

[0041] For example, a HIVE table stores employee information. The table name is staff and the table fields are as follows:

[0042]

[0043]

[0044] Extract the variable name of the script string (for example, age and level in "age>10and level>20"), find the value that matches the variable name in the field name array (for example, the table name staff), and record its position in the array. For example, ["age", "level"].

[0045] Take out the value at the same position in the field value array, such as [5,16], and then replace the variable in the expression with the taken value. For example, the script string becomes "5>10and 16>20". Load and execute the replaced script string. If the return value is 1, it means that the rules are met. If the return value is 0, it means that the rules are not met.

[0046] It is worth noting that before reading the source data in the storage medium (such as HDFS) and inputting it into the script, you can write a HIVE script to query the source data in HDFS. Of course, when writing a HIVE script to query the source data in HDFS, you can set limiting conditions, for example: query employees who joined the company on or after January 1, 2019 and whose age and level meet certain conditions. HDFS is the Hadoop distributed file system.

[0047] It can be seen that the present invention combines the UDF function with HIVE SQL. HIVE SQL first screens out the data set that needs to be filtered, and each data in the data set is used as an input parameter of the UDF function. The UDF function then gives whether each data meets the condition, for example: if it meets the condition, 1 is returned, and if it does not meet the condition, 0 is returned.

[0048] In other embodiments, the script is started by the custom function to execute Groovy to determine whether the source data meets the preset conditions. The specific implementation process includes:

[0049] The script is executed by the custom function, and the execution result is obtained and stored in the corresponding position of the source data in the HIVE table. According to the execution result, it is determined whether the source data meets the preset conditions.

[0050] like Figure 2 As shown, it is a framework diagram of the method for data processing according to an embodiment of the present invention, including rule configuration and rule execution. Wherein, when configuring the rules, the page content can be read by JavaScript based on the rules of the page configuration, parsed into a specific script string, and then the script string is stored in the HIVE table. When the rules are executed, the script string can be taken out from the HIVE table by the expression loader, and according to the variable name in the script string, the value consistent with the variable name is found in the field name array (HIVE table), and the position of the value in the field name array is recorded. The numerical value is taken out at the same position in the field value array (HIVE table) to replace the variable in the script string. Then the replaced script string is executed, and the source data is determined to meet the preset conditions according to the return value (i.e., the expression result), and the final data is output according to the result of the rule filtering. Of course, the return value can be stored in the corresponding position in the HIVE table.

[0051] Figure 3 is a schematic diagram of the main flow of a data processing method according to another embodiment of the present invention. The data processing method may include:

[0052] Step S301, reading page configuration content through JavaScript.

[0053] Step S302: parsing into a specific Groovy script string based on the page configuration rules.

[0054] Among them, Groovy is an agile dynamic language for Java virtual machines. It is a mature object-oriented programming language that can be used for object-oriented programming and as a pure scripting language. Groovy scripts mainly implement the combination of arithmetic expressions and logical expressions, and are responsible for determining whether each rule is valid and the relationship between multiple rules.

[0055] Step S303: store the Groovy script string into a HIVE table.

[0056] Step S304: Write a UDF function in Hadoop using Java code to generate a custom function.

[0057] Step S305, reading the HIVE table through a custom function, taking out the Groovy script string, and loading the Groovy script string.

[0058] Step S306, writing a HIVE script to query the source data in the HDFS, so as to read the source data in the HDFS and input it into the Groovy script.

[0059] In some embodiments, the specific implementation process of reading source data in HDFS and inputting it into a Groovy script includes: according to the variable name in the Groovy script string, finding a value consistent with the variable name in the field name array, recording the position of the value in the field name array, and taking the value at the same position in the field value array to replace the variable in the script string.

[0060] Step S307, starting Groovy to execute the script through the custom function, determining whether the source data meets the preset conditions, so as to perform the screening of the source data. The specific implementation process includes:

[0061] The Groovy script is executed by the custom function to obtain the execution result and store it in the corresponding position of the source data in the HIVE table. According to the execution result, it is determined whether the source data meets the preset conditions. For example, if the execution result is 1, it means that the condition is met, and if the execution result is 0, it means that the condition is not met.

[0062] In summary, the data processing method proposed in the present invention configures and generates arithmetic and logical expression strings, loads the strings through Groovy and then runs the expressions. At the same time, the results can be combined with HIVE (Hadoop) to process massive data. In addition, rule changes do not require code changes, and direct page configuration changes can take effect immediately, that is, Groovy loads and runs scripts that are parsed instantly, so only the expression needs to be updated through the page configuration, and the rules can take effect immediately without the need to rewrite the Hadoop script.

[0063] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention, such as Figure 4 As shown, the data processing device 400 includes an acquisition module 401 and a processing module 402. The acquisition module 401 acquires the page configuration content, parses it into a script string based on the page configuration rules, and then loads the script string through a preset custom function. The processing module 402 is used to read the source data and input it into the script, execute the script through the custom function, determine whether the source data meets the preset conditions, and perform screening of the source data.

[0064] In some embodiments, the acquisition module 401 parses the script string based on the rules configured on the page, including:

[0065] Read the page configuration content through JavaScript, splice it into a specific expression string, and store the expression string in the HIVE table.

[0066] In addition, in some embodiments, before the acquisition module 401 loads the script string through a preset custom function, it includes:

[0067] Write UDF functions in Hadoop to generate custom functions.

[0068] In some embodiments, the acquisition module 401 loads the script string through a preset custom function, including:

[0069] Read the HIVE table through the preset custom function, extract the script string, and then load the script string through Groovy.

[0070] In some embodiments, the processing module 402 reads source data and inputs it into a script, including:

[0071] According to the variable name in the script string, find the value consistent with the variable name in the field name array, and record the position of the value in the field name array;

[0072] Replace the variable in the script string with the value at the same position in the field value array.

[0073] In some embodiments, the processing module 402 executes the script through the custom function to determine whether the source data meets the preset conditions, including:

[0074] Start Groovy to execute the script through the custom function, obtain the execution result and store it in the corresponding position of the source data in the HIVE table;

[0075] Based on the execution result, determine whether the source data meets the preset conditions.

[0076] In some embodiments, before the processing module 402 reads the source data and inputs it into the script, it includes:

[0077] Write HIVE scripts to query source data in HDFS.

[0078] It should be noted that the data processing method and the data processing device described in the present invention have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0079] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0080] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0081] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0082] The terminal devices 501 , 502 , and 503 may be various electronic devices having a screen for data processing and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0083] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0084] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the computing device is generally arranged in the server 505 .

[0085] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0086] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0087] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal data processor (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0089] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0090] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, it may be described as follows: a processor includes an acquisition module and a processing module. The names of these modules do not, in some cases, limit the modules themselves.

[0093] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes obtaining page configuration content, parsing it into a script string based on the page configuration rules, and then loading the script string through a preset custom function; reading source data and inputting it into a script, executing the script through the custom function, and determining whether the source data meets the preset conditions to perform screening of the source data.

[0094] The technical solution according to the embodiment of the present invention can solve the problem of poor timeliness of existing mass data filtering.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Obtaining page configuration content, parsing it into a script string based on page configuration rules, and then loading the script string through a preset custom function; Reading source data and inputting it into the script includes: searching for a value consistent with the variable name in the field name array according to the variable name in the script string, and recording the position of the value in the field name array; taking out the value at the same position in the field value array to replace the variable in the script string; The script is executed through the user-defined function to determine whether the source data meets the preset conditions, so as to perform screening on the source data.

2. The method according to claim 1, characterized in that Parsed into script strings based on page configuration rules, including: Read the page configuration content through JavaScript, splice it into a specific expression string, and store the expression string in the HIVE table.

3. The method according to claim 1, characterized in that Before loading the script string through a preset custom function, include: Write UDF functions in Hadoop to generate custom functions.

4. The method according to claim 1, characterized in that: The script string is loaded through a preset custom function, including: Read the HIVE table through the preset custom function, extract the script string, and then load the script string through Groovy.

5. The method according to claim 1, characterized in that Executing the script through the custom function to determine whether the source data meets the preset conditions includes: Start Groovy to execute the script through the custom function, obtain the execution result and store it in the corresponding position of the source data in the HIVE table; Based on the execution result, determine whether the source data meets the preset conditions.

6. The method according to any one of claims 1 to 5, characterized in that: Read source data before inputting it into the script, including: Write HIVE scripts to query source data in HDFS.

7. A data processing device, characterized in that: include: An acquisition module is used to acquire page configuration content, parse it into a script string based on page configuration rules, and then load the script string through a preset custom function; The processing module is used to read the source data and input it into the script, including: according to the variable name in the script string, find the value consistent with the variable name in the field name array, and record the position of the value in the field name array; take out the value at the same position in the field value array to replace the variable in the script string; execute the script through the custom function to determine whether the source data meets the preset conditions to perform the screening of the source data.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images