Audit method based on HDFS database, electronic equipment, medium and program product
By parsing operation messages of the HDFS database to generate audit logs, the security issues of HDFS database files are resolved, high-risk operations are identified and blocked, and a secure data environment and detailed operation and maintenance support are provided.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-04-03
AI Technical Summary
File security issues in HDFS databases have not been effectively resolved, especially in high-throughput and distributed environments, where there is a lack of effective auditing mechanisms.
By parsing the operation request and response messages of the HDFS database, audit logs are generated, file operation types and message address information are identified, and rules are configured to block high-risk operations, providing detailed operation and maintenance logs.
It enables full auditing of HDFS database operations, can identify and block high-risk operations, provides a secure data environment, and provides detailed log support for operations and maintenance personnel.
Smart Images
Figure CN121786009A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of HDFS (Hadoop Distributed File System) technology, and in particular to an auditing method, device, media, and program product based on an HDFS database. Background Technology
[0002] With the rapid development of the information age and the emergence of various file systems, HDFS, as a highly fault-tolerant distributed file system suitable for deployment on inexpensive machines and providing high-throughput data access, is being used more and more widely, making it ideal for applications with extremely large datasets. However, file security in HDFS has always been a concern. Summary of the Invention
[0003] The purpose of this invention is to provide an auditing method, device, medium, and program product based on an HDFS database, so as to at least partially solve the above-mentioned problems of the prior art.
[0004] To achieve the above objectives, one aspect of the present invention provides an auditing method based on an HDFS database, comprising: Retrieve request messages for operations on the HDFS database; Based on the preset correspondence between identifier information and fields, the request line, request body and status line of the request message are parsed to obtain the parsing results including message address information and file operation type; Parse the response message corresponding to the request message, and generate an audit log for the operation based on the parsing results of the request message and the response message.
[0005] Preferably, the step of parsing the request line and request body status line of the request message according to the preset correspondence between identifier information and fields includes: The request line is identified based on the keyword prefix and the URL; The request body includes one or more of the following identification information: Host for identifying the domain name, Connection for identifying the connection, Content-Length for identifying the length of the request body, Content-Type for identifying the data type of the request, User-Agent for identifying the user's operating system and browser version, and Referer for identifying which page the current page was redirected from. Identify blank lines in the request message; The status line is identified and parsed after the empty line.
[0006] Preferably, parsing the request line of the request message includes: Parse the header features, database name, file operation type, and username in the request line; wherein, the header feature identification information includes one or more of the following keywords: GET, PUT, UPDATE, DELETE; the database name identification information includes webhdfs; the username identification information includes hadoop; the file operation type identification information includes one or more of the following: CREATE, OPEN, DELETE, RENAME, MKDIRS, LISTSTATUS, GETFILESTATUS, where CREATE indicates creating a new file, OPEN indicates opening a file, DELETE indicates deleting a file, RENAME indicates renaming, MKDIRS indicates creating a new folder, LISTSTATUS indicates viewing a file list, and GETFILESTATUS indicates viewing a file.
[0007] Preferably, parsing the response message corresponding to the request message includes: Identify the response header, blank line, and response body of the response message, wherein the response body is located after the blank line, and the response header includes one or more of the following identification information: Content-Type for identifying the format of the response body, Expires for identifying the cache expiration time, Set-Cookie for identifying the cache, Content-Length for identifying the length of the response body, and Server for identifying the software name and version number of the Web service.
[0008] Preferably, parsing the response message corresponding to the request message includes: Determine whether the response message is fragmented; If the judgment result is yes, the fragmented packets are cached and parsed only after the complete packet is obtained.
[0009] Preferably, the request message for obtaining operations on the HDFS database includes: Obtain communication messages from the HDFS database. If the data direction in the message is upward, determine that the message is the request message; if the data direction in the message is downward, determine that the message is the response message.
[0010] Preferably, before obtaining the request message for operations on the HDFS database, the method further includes: Configure the HDFS Explorer client to enable the WebHDFS service.
[0011] Another aspect of the present invention provides an electronic device, the electronic device comprising: processor; Memory used to store processor-executable instructions; When the processor invokes the executable instructions, it causes the electronic device to implement the auditing method based on the HDFS database provided by the above aspects and any of them.
[0012] Another aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the auditing method based on an HDFS database provided in the above aspects and any of the above methods.
[0013] Another aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the auditing method based on an HDFS database provided in the above aspects and any of the above methods.
[0014] Compared with the prior art, the present invention has at least the following advantages: By parsing the request and response messages of HDFS database operations, the system can analyze database-related operations in a complete session, output audit logs, and obtain message address and file operation type information. This enables auditing of HDFS database operations and allows for the configuration of targeted rules to alert, intercept, and block high-risk operations that are of great concern to users, providing users with a more secure data environment and offering operations and maintenance personnel the most intuitive and detailed log information. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating an auditing method based on an HDFS database, provided as an embodiment of the present invention.
[0016] Figure 2 This is an example of parsing a request message in the method provided in the embodiments of the present invention.
[0017] Figure 3 This is an example of parsing a response message in the method provided in this embodiment of the invention.
[0018] Figure 4 This is a schematic diagram of an auditing device based on an HDFS database provided in an embodiment of the present invention.
[0019] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate to understand the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a product or device comprising a series of units is not necessarily limited to those explicitly listed, but may include other units not explicitly listed or inherent to such product or device.
[0022] In this invention, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing the invention and its embodiments, and are not intended to limit the indicated devices, elements, or components to having a specific orientation, or to be constructed and operated in a specific orientation.
[0023] Furthermore, in addition to indicating direction or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in certain situations to indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.
[0024] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.
[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Example
[0026] Embodiment 1 of this invention provides an auditing method based on an HDFS database. This method audits the communication content of the HDFS database, primarily covering various operations such as creating folders, viewing file lists, viewing files, opening files, deleting files, renaming files, uploading files, downloading files, and cutting files. It can audit all database-related operations throughout the entire connection from the start to the end of the session, ultimately displaying the results as audit logs. Figure 1 A flowchart illustrating this method is shown. (For example...) Figure 1 As shown, the method includes: Step 101: Obtain the request message for operations on the HDFS database.
[0027] In this step, the auditing system can obtain communication messages from the HDFS database. If the data direction in the message is upstream, the message is determined to be a request message. If the data direction in the message is downstream, the message is determined to be a response message.
[0028] Additionally, before obtaining the request message for operations on the HDFS database in this step, you can also include configuring the HDFS Explorer client to open the WebHDFS service. For example, when using the HDFS Explorer client to access the HDFS database on Windows, you need to configure and open the WebHDFS service first. Specifically, you need to add the following configuration to hdfs-site.xml to log in and access it normally.
[0029] <configuration> <property> <name> dfs.webhdfs.enabled< / name> <value> true< / value> < / property> Step 102: Based on the preset correspondence between the identifier information and the fields, parse the request line, request body, and status line of the request message to obtain the parsing results, including message address information and file operation type.
[0030] In a preferred embodiment, this step includes: Identify the request line based on the keyword at the beginning and the URL; The parsing request body includes one or more of the following identifying information: Host for identifying the domain name, Connection for identifying the connection, Content-Length for identifying the length of the request body, Content-Type for identifying the data type of the request, User-Agent for identifying the user's operating system and browser version, and Referer for identifying which page the current page was redirected from. Identify blank lines in the request message; Identify the status line after a blank line.
[0031] The request line for parsing the request message may include: Parse the header characteristics, database name, file operation type, and username in the request line. The header characteristics include one or more of the following keywords: GET, PUT, UPDATE, DELETE; the database name includes webhdfs; the username includes hadoop; and the file operation type includes one or more of the following: CREATE, OPEN, DELETE, RENAME, MKDIRS, LISTSTATUS, GETFILESTATUS, where CREATE indicates creating a new file, OPEN indicates opening a file, DELETE indicates deleting a file, RENAME indicates renaming, MKDIRS indicates creating a new folder, LISTSTATUS indicates viewing a file list, and GETFILESTATUS indicates viewing a file.
[0032] Since most file operations involve at least two combined operation types, the actual file operation types for operations like uploading a file are: OPEN + CREATE, renaming is: RENAME + DELETE, copying is: OPEN + CREATE, and cutting is: OPEN + CREATE + DELETE.
[0033] Figure 2 This shows an example of parsing a request message. Figure 2 The request message shown includes a request line, request body, blank line, and status line. The request line contains the header features GET, the database name webhdfs, the operation type LISTSTATUS, and the username hadoop. In the request body, Host identifies the domain name, followed by the domain value 10.55.65.221:50070; Connection identifies the connection, followed by the value keep-alive, indicating that the request supports persistent connections; Content-Length indicates the length of the request body, and can be followed by a specific length value; Content-Type identifies the data type of the request, and can be followed by a specific type value; User-Agent identifies the user's operating system and browser version, and can be followed by the specific values for the user's operating system and browser version; Referer identifies which page the current page was redirected from, and can be followed by the specific page address information. A blank line follows the request body. The status line follows the blank line. Figure 2 The status line shown is HTTP / 1.1 200 OK. The status line follows a blank line. If content similar to HTTP / 1.1 200 OK appears before a blank line, the auditing system will not recognize it as a status line.
[0034] Step 103: Parse the response message corresponding to the request message, and generate an audit log for the operation based on the parsing results of the request message and the response message.
[0035] The response messages corresponding to the parsing request messages include: Identify the response header, blank line, and response body of the response message. The response body is located after the blank line. The response header includes one or more of the following identification information: Content-Type for identifying the format of the response body, Expires for identifying the cache expiration time, Set-Cookie for identifying the cache, Content-Length for identifying the length of the response body, and Server for identifying the software name and version number of the web service.
[0036] Figure 3 This shows an example of parsing a response message. Figure 3 The response message shown includes a response header, a blank line, and a response body. The response header includes identification information such as Content-Type, Expires, Set-Cookie, Content-Length, and Server. The values or content following these indicate the specific response body format, cache, response body length, and the software name and version number of the web service, respectively. A blank line follows the response header, and then the response body. The format of the response body is not fixed and depends mainly on the type of resource requested and the server configuration; it typically includes HTML, JSON, images, etc.
[0037] In a preferred embodiment, parsing the response message corresponding to the request message may include: determining whether the response message is fragmented; if the determination result is yes, caching the fragmented message, and then parsing it after obtaining the complete message. In other words, the parsing of the response message needs to consider the case of fragmented packets. At the response processing entry point, it is necessary to determine whether fragmentation exists. Fragmented messages should be cached first, and after caching, the return code and other content such as return results and error messages in the response body should be parsed according to the actual data length and the complete data. Upon completion of response processing, a corresponding event processing end marker will be generated, and the corresponding event count will be performed.
[0038] Once a request and its corresponding response have been parsed, a complete event parsing is completed. The final result can be generated, which includes information such as source IP, source port, destination IP, destination port, username, database name, file operation type, request processing URL, return code, error message, and return result. This result is then output as an audit log.
[0039] By employing the method provided in this embodiment of the invention, request messages and response messages of HDFS database operations are parsed, enabling the output of audit logs after parsing database-related operations in a complete session, and obtaining message address information and file operation type information. This allows for the auditing of HDFS database operations, and allows for the targeted configuration of rules to alert, intercept, and block high-risk operations that are of great concern to users, providing users with a more secure data environment and providing operation and maintenance personnel with the most intuitive and detailed log details. Example
[0040] Based on the same technical concept as Embodiment 1, this embodiment of the invention provides an auditing device based on an HDFS database. The specific implementation of this device can be referred to the description in Embodiment 1 above. Figure 4 A schematic diagram of the device is shown, as follows: Figure 4 As shown, the device includes: The message acquisition unit 41 is used to acquire request messages for operations on the HDFS database; The parsing unit 42 is used to parse the request line, request body, and status line of the request message according to the preset correspondence between the identification information and the fields, and obtain the parsing result including message address information and file operation type; and parse the response message corresponding to the request message. The log generation unit 43 is used to generate an audit log for the operation based on the parsing results of the request message and the parsing results of the response message.
[0041] In a preferred embodiment, the parsing unit 42 can be used for: The request line is identified based on the keyword prefix and the URL; The request body includes one or more of the following identification information: Host for identifying the domain name, Connection for identifying the connection, Content-Length for identifying the length of the request body, Content-Type for identifying the data type of the request, User-Agent for identifying the user's operating system and browser version, and Referer for identifying which page the current page was redirected from. Identify blank lines in the request message; The status line is identified and parsed after the empty line.
[0042] In a preferred embodiment, the parsing unit 42 can be used for: Parse the header features, database name, file operation type, and username in the request line; wherein, the header feature identification information includes one or more of the following keywords: GET, PUT, UPDATE, DELETE; the database name identification information includes webhdfs; the username identification information includes hadoop; the file operation type identification information includes one or more of the following: CREATE, OPEN, DELETE, RENAME, MKDIRS, LISTSTATUS, GETFILESTATUS, where CREATE indicates creating a new file, OPEN indicates opening a file, DELETE indicates deleting a file, RENAME indicates renaming, MKDIRS indicates creating a new folder, LISTSTATUS indicates viewing a file list, and GETFILESTATUS indicates viewing a file.
[0043] In a preferred embodiment, the parsing unit 42 can be used for: Identify the response header, blank line, and response body of the response message, wherein the response body is located after the blank line, and the response header includes one or more of the following identification information: Content-Type for identifying the format of the response body, Expires for identifying the cache expiration time, Set-Cookie for identifying the cache, Content-Length for identifying the length of the response body, and Server for identifying the software name and version number of the Web service.
[0044] In a preferred embodiment, the parsing unit 42 can be used for: Determine whether the response message is fragmented; If the judgment result is yes, the fragmented packets are cached and parsed only after the complete packet is obtained.
[0045] The acquisition unit 41 can be used for: Obtain communication messages from the HDFS database. If the data direction in the message is upward, determine that the message is the request message; if the data direction in the message is downward, determine that the message is the response message.
[0046] In a preferred embodiment, the device may further include: a configuration unit for configuring the HDFS Explorer client to open the webhdfs service. Example
[0047] Based on the same technical concept as Embodiment 1, this embodiment of the invention provides an electronic device, with reference to... Figure 5 As shown, the electronic device includes: a processor 51 and a memory 52 for storing processor-executable instructions; wherein, when the processor 51 invokes the executable instructions, the electronic device implements the auditing method based on the HDFS database provided in Embodiment 1 and any of its embodiments.
[0048] The electronic device may also include input interfaces, output interfaces, buses, and other components. The input and output interfaces are used for communication between the electronic device and external devices. The bus is used for communication between internal components such as processors and memory. Example
[0049] Based on the same technical concept as Embodiment 1, this embodiment of the invention provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the auditing method based on the HDFS database provided above and in any manner thereof. Example
[0050] Based on the same technical concept as Embodiment 1, this embodiment of the invention provides a computer program product, which includes a computer program that, when executed by a processor, implements the auditing method based on the HDFS database provided by the above aspects and any of them.
[0051] By adopting the solution provided in this embodiment of the invention, the request messages and response messages of HDFS database operations are parsed, enabling the output of audit logs after parsing database-related operations in a complete session, and obtaining message address information and file operation type information. This allows for the auditing of HDFS database operations, and allows for the targeted configuration of rules to alert, intercept, and block high-risk operations that are of great concern to users, providing users with a more secure data environment and providing operation and maintenance personnel with the most intuitive and detailed log details.
[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / configuration>
Claims
1. An auditing method based on an HDFS database, characterized in that, include: Retrieve request messages for operations on the HDFS database; Based on the preset correspondence between identifier information and fields, the request line, request body and status line of the request message are parsed to obtain the parsing results including message address information and file operation type; Parse the response message corresponding to the request message, and generate an audit log for the operation based on the parsing results of the request message and the response message.
2. The auditing method based on HDFS database according to claim 1, characterized in that, The step of parsing the request line and request body status line of the request message according to the preset correspondence between identifier information and fields includes: The request line is identified based on the keyword prefix and the URL; The request body includes one or more of the following identification information: Host for identifying the domain name, Connection for identifying the connection, Content-Length for identifying the length of the request body, Content-Type for identifying the data type of the request, User-Agent for identifying the user's operating system and browser version, and Referer for identifying which page the current page was redirected from. Identify blank lines in the request message; The status line is identified and parsed after the empty line.
3. The auditing method based on HDFS database according to claim 1, characterized in that, The request line of the parsed request message includes: Parse the header features, database name, file operation type, and username in the request line; wherein, the header feature identification information includes one or more of the following keywords: GET, PUT, UPDATE, DELETE; the database name identification information includes webhdfs; the username identification information includes hadoop; the file operation type identification information includes one or more of the following: CREATE, OPEN, DELETE, RENAME, MKDIRS, LISTSTATUS, GETFILESTATUS, where CREATE indicates creating a new file, OPEN indicates opening a file, DELETE indicates deleting a file, RENAME indicates renaming, MKDIRS indicates creating a new folder, LISTSTATUS indicates viewing a file list, and GETFILESTATUS indicates viewing a file.
4. The auditing method based on an HDFS database according to any one of claims 1-3, characterized in that, The parsing of the response message corresponding to the request message includes: Identify the response header, blank line, and response body of the response message, wherein the response body is located after the blank line, and the response header includes one or more of the following identification information: Content-Type for identifying the format of the response body, Expires for identifying the cache expiration time, Set-Cookie for identifying the cache, Content-Length for identifying the length of the response body, and Server for identifying the software name and version number of the Web service.
5. The auditing method based on HDFS database according to claim 1, characterized in that, The parsing of the response message corresponding to the request message includes: Determine whether the response message is fragmented; If the judgment result is yes, the fragmented packets are cached and parsed only after the complete packet is obtained.
6. The auditing method based on an HDFS database according to any one of claims 1-3, characterized in that, The request message for obtaining operations on the HDFS database includes: Obtain communication messages from the HDFS database. If the data direction in the message is upward, determine that the message is the request message; if the data direction in the message is downward, determine that the message is the response message.
7. The auditing method based on an HDFS database according to any one of claims 1-3, characterized in that, Before obtaining the request message for operations on the HDFS database, the following are also included: Configure the HDFS Explorer client to enable the WebHDFS service.
8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; When the processor invokes the executable instructions, it causes the electronic device to implement the auditing method based on the HDFS database according to any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the auditing method based on an HDFS database according to any one of claims 1-7.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the auditing method based on an HDFS database according to any one of claims 1-7.