Data processing method and device, nonvolatile storage medium and computer equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and computer equipment. The method comprises the steps that initial data are received, and the initial data are obtained by conducting data collection on a plurality of subsystems in a network system through a collector; user data in the network system is analyzed from the initial data, and the user data comprises a user identifier and a user location; associating the user data with other data to obtain target data, the other data being data corresponding to the user data in the initial data; and storing the target data into a preset database. According to the invention, the technical problem that the timely identification and response to security threats are affected due to the fact that a plurality of different systems possibly exist in the internal network environment of an enterprise at present and data stored among the systems are mutually independent and are difficult to quickly and accurately analyze user behaviors is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data processing method, apparatus, non-volatile storage medium, and computer device. Background Art

[0002] Currently, there are multiple systems in most enterprises. However, these systems often come from different R & D teams or different software suppliers, and each system has its own management focus, such as account system permission management, network resource permission management, intranet data transmission auditing, or employee system operation record auditing. The data between systems is independent of each other, and the definitions of users are different, so a comprehensive employee behavior record cannot be formed. This makes it difficult for security operation personnel to quickly and accurately obtain all the behavior information of employees in the intranet, thus affecting the timely identification and response to security threats.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, apparatus, non-volatile storage medium, and computer device to at least solve the technical problem that there may be multiple different systems in the enterprise internal network environment currently, and the data stored between each system is independent of each other, resulting in difficulty in quickly and accurately analyzing user behavior, thus affecting the timely identification and response to security threats.

[0005] According to one aspect of the embodiments of the present invention, a data processing method is provided, including: receiving initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; parsing user data in the network system from the initial data, where the user data includes a user identifier and a user location; associating the user data with other data to obtain target data, where the other data is data corresponding to the user data in the initial data; and storing the target data in a preset database.

[0006] Optionally, receiving the initial data includes: receiving encrypted data in the collector, where the encrypted data is data encrypted based on the TLS protocol; verifying the certificate of the collector; and decrypting the encrypted data based on the TLS protocol to obtain the initial data when the verification passes.

[0007] Optionally, parsing user data in the network system from the initial data includes: dividing the initial data into multiple types of data based on a preset configuration tag; and respectively parsing the multiple types of data using the corresponding parsing methods for each type to obtain the user data.

[0008] Optionally, there are multiple types including access log type, application platform operation log type, application platform operation alarm type, data leakage prevention system log type, and data leakage prevention system alarm type. Among them, the data of the access log type includes user identification, user location, user MAC address, and log information. The data of the application platform operation log type includes the application platform name, user platform name, and corresponding log information. The data of the application platform operation alarm type includes the application platform name, user platform name, and corresponding alarm information. The data of the data leakage prevention system log type includes the user MAC address and corresponding log information. The data of the data leakage prevention system alarm type includes the user MAC address and corresponding alarm information.

[0009] Optionally, use the parsing methods corresponding to each of the multiple types to parse the data of the multiple types respectively to obtain user data, including: in the case where the target data is of the data leakage prevention system log type or the data leakage prevention system alarm type, determine the target user MAC address in the target data; based on the data corresponding to the access log, determine the first correspondence between the user MAC address and the user data; based on the first correspondence, select the user data corresponding to the target user MAC address as the target user data corresponding to the target data.

[0010] Optionally, use the parsing methods corresponding to each of the multiple types to parse the data of the multiple types respectively to obtain user data, including: in the case where the target data is of the application platform operation log type or the application platform operation alarm type, extract the target application platform name and the target application platform user name in the target data; obtain the target user identification corresponding to the target application platform user name in the target application platform; based on the data corresponding to the access log, determine the second correspondence between the user identification and the user location; based on the second correspondence, select the user location corresponding to the target user identification as the target user location corresponding to the target data; according to the target user identification and the target user location, determine the target user data corresponding to the target data.

[0011] Optionally, the collector is a Fluentd collector.

[0012] According to another aspect of the embodiments of the present invention, there is also provided a data processing device, including: a receiving module, configured to receive initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; a parsing module, configured to parse user data in the network system from the initial data, where the user data includes user identification and user location; an association module, configured to associate the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; a storage module, configured to store the target data in a preset database.

[0013] According to another aspect of the embodiments of the present invention, a non-volatile storage medium is further provided. The non-volatile storage medium includes a stored program, wherein when the program runs, it controls the device where the non-volatile storage medium is located to execute any one of the above data processing methods.

[0014] According to still another aspect of the embodiments of the present invention, a computer device is further provided. The computer device includes a processor for running a program, wherein when the program runs, it executes any one of the above data processing methods.

[0015] In the embodiments of the present invention, by adopting the data processing method, initial data is received, wherein the initial data is obtained by a collector collecting data from multiple subsystems in a network system; user data in the network system is parsed from the initial data, wherein the user data includes a user identifier and a user's place of residence; the user data is associated with other data to obtain target data, wherein the other data is the data corresponding to the user data in the initial data; and the target data is stored in a preset database, achieving the purpose of summarizing and storing user data for facilitating user behavior analysis, thereby realizing the technical effects of improving the response speed to security threats and enhancing the security of the system, and further solving the technical problem that there may be multiple different systems in the current enterprise internal network environment, and the data stored between each system is independent of each other, making it difficult to quickly and accurately analyze user behavior, thus affecting the timely identification and response to security threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing the data processing method is shown;

[0018] Figure 2 It is a flowchart of the data processing method provided by the embodiments of the present invention;

[0019] Figure 3 It is a schematic diagram of multi-location collection of the data processing method provided by an optional embodiment of the present invention;

[0020] Figure 4 It is a block diagram of the structure of the data processing device provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0024] Flink: Apache Flink is a framework and distributed processing engine for performing stateful computations on unbounded and bounded data streams. Flink can run in all common cluster environments and can perform computations at in-memory speed and at any scale.

[0025] Fluentd: A cross-platform and extensible log collector whose main function is to collect, transform, and send logs. It can collect logs from various sources (such as files, TCP / UDP, Syslog, Apache, etc.) and can send these logs to various destinations (such as Elasticsearch, Hadoop, MongoDB, etc.).

[0026] Admission system: A system that provides network access management for visitors and sets access control permissions according to different roles.

[0027] 4A system: Refers to a unified security management platform for authentication, authorization, account, and audit.

[0028] Data Leakage Prevention System (DLP): Data leakage prevention is a data security management system that uses data leakage prevention (DLP) technical means to prevent specified data or information assets of an enterprise from flowing out of the enterprise in a form that violates the security policy regulations.

[0029] User Behavior Audit System: A system that records and collects account operation behaviors of the digital application platforms within an enterprise and conducts operation security analysis for each application platform account.

[0030] According to an embodiment of the present invention, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0031] The method embodiment provided in the first embodiment of this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing the data processing method is shown. As Figure 1 shown, the computer terminal 10 may include one or more (shown as 102a, 102b,..., 102n in the figure) processors (the processor may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0032] It should be noted that the above one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this article. The data processing circuit can be embodied in software, hardware, firmware, or any combination thereof in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10. As involved in the embodiments of this application, the data processing circuit is a processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the data processing method of the above application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0034] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computer terminal 10.

[0035] Figure 2 is a schematic flowchart of the data processing method provided according to the embodiments of the present invention, as Figure 2 shown, the method includes the following steps:

[0036] Step S202, receiving initial data, where the initial data is obtained by the collector collecting data from multiple subsystems in the network system.

[0037] In this step, the collector can be deployed in the office network or internal computer room in each region of the enterprise to receive data in the environment of each office network and internal network. Among them, the data can be various system logs and alarm data. For example, an enterprise has multiple independent internal network office areas A1, A2,... Each area is deployed with a collector, and the collector in each area starts a collection process to receive the operation behavior records of the application platform accounts of the business system, the log records of the access control system, and the logs and alarm records of the DLP system. The execution subject of this embodiment can be a collector in a secure independent network environment, and this collector is used to collect the data collected by the collectors in multiple regions. For example, the network environment where the security data analysis system is located is A0, and A0 is deployed with collector nodes for receiving data from each computer room. The initial data can be received through a secure channel, which can also improve the security of data transmission. Among them, the security data analysis system can be used to perform user behavior data analysis.

[0038] Step S204, parsing user data in the network system from the initial data, where the user data includes a user identifier and a user's territorial location.

[0039] In this step, user data in the network system can be parsed from the received initial data, especially extracting user identifiers (such as 4A accounts) and user territorial information. Extracting user data allows other data to be associated with user data in subsequent processes, facilitating more convenient analysis of the behavior data of different users and thereby determining security threats. For example, the initial data can be parsed to extract key fields. During the parsing process, fields can be mapped to ensure that all user identifiers and territorial information provided by the systems are converted into a unified format. For instance, if the user identifier in the 4A system is the "user" field while other systems use the "username" field, the parsing process needs to unify these fields into a standard field name, such as "userid". Standardize the extracted fields, including data cleaning, formatting, and possible conversions, to ensure data consistency and accuracy. The processed data can be used as the input to a stream processing framework (such as Apache Flink). In Flink, the DataStream API can be used to process stream data, and operators (such as map, filter, flatMap, etc.) can be used to further process and transform the data. In the stream processing pipeline, operators are used to extract the user identifier and user territorial information from the user data.

[0040] Through the above steps, the user data parsed from the initial data not only contains the user's identifier but also the user's current territorial information, which helps to construct a more comprehensive view of employee behavior and is crucial for identifying and preventing internal security threats.

[0041] Step S206: Associate the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data.

[0042] In this step, user data is associated with other data to obtain target data. This step usually involves using data stream join or map operators in a streaming computing framework to integrate user data containing user identification and user location information with data from other systems. For example, in Flink or other streaming computing frameworks, the join operator can be used to associate the user data stream with other data streams. The join operator allows records in the data stream to be matched based on one or more fields. For example, the DLP log stream containing the mac address can be joined with the macStream stream containing the user and mac relationship to identify the employee to whom the log record belongs. During the data stream join process, a state backend (such as MapState or ListState) is used to store temporary mapping relationships, such as mac->user and user->location. State management is the cornerstone of data association in stream computing, enabling each operator instance to store and query these relationships to achieve real-time data integration. After joining and associating, the resulting data stream contains user identification, user location, and other key information, such as event type, time, etc. This information together constitutes the target data, that is, a panoramic record of employees' security behaviors.

[0043] Specifically, there is usually a 4A system within the company, which usually includes the basic information and permissions of users. Among them, the user identification can be the 4A account of the user. Therefore, associating user data with other data can also be regarded as associating the data of each system with the employee accounts in the 4A system.

[0044] Through these steps, user data can be associated with data from other subsystems in real time to form target data, that is, a record containing complete employee behavior information, providing a basis for subsequent security analysis and behavior monitoring.

[0045] Step S208, store the target data in a preset database.

[0046] In this step, the processed and analyzed target data, that is, the log and alarm data that have been integrated and marked with employee information and location information, needs to be stored in the database for subsequent queries, reports, alarms, and long-term data analysis. The associated data all carry employee information and office locations, thus forming a complete log and alarm data record for the employee. Write alarms and logs to a unified alarm table and log table respectively. Before storing the target data in the database, ensure that the data format matches the database table structure. This may involve defining appropriate field types such as strings, dates and times, integers, etc., and ensuring data integrity. For example, ensure that all log and alarm records contain fields such as user, mac, location, sourceType, and message. In a Flink job, you can configure the Sink operator to write the processed data to the database. This can be achieved by using the Apache Flink pre-set database connection plug-in or a custom Sink function. Choose batch or real-time data writing mode according to business needs and database characteristics. Batch mode is usually used for writing large data sets, which can reduce the frequency of database writes and improve write efficiency. Real-time writing has high requirements on the real-time performance of data and is suitable for scenarios that require immediate response.

[0047] Through the above steps, it can be ensured that the target data after streaming computing analysis is effectively stored in the preset database, providing a persistent data foundation for the company's data security analysis, compliance inspection, and employee behavior monitoring. This not only helps with real-time security response, but also provides a basis for long-term data analysis and security management strategies.

[0048] Through the above steps, the purpose of aggregating and storing user data to facilitate user behavior analysis can be achieved, thereby realizing the technical effect of improving the response speed of security threats and improving the security of the system, and further solving the technical problem that there may be multiple different systems in the current enterprise internal network environment, and the data stored in each system are independent of each other, resulting in difficulty in quickly and accurately analyzing user behavior, thereby affecting the timely identification and response to security threats.

[0049] As an optional embodiment, receiving initial data includes: receiving encrypted data in a collector, wherein the encrypted data is data encrypted based on the TLS protocol; verifying the certificate of the collector; and if the verification passes, decrypting the encrypted data based on the TLS protocol to obtain the initial data.

[0050] Optionally, during the data transmission process, the data can be encrypted based on the TLS protocol to ensure data security and prevent malicious attacks and leakage. Specifically, the TLS parameters can be set in the configuration file of the receiving end (such as the collector in a secure environment) to establish a secure connection. These parameters include the certificate path, private key path, password phrase of the private key, etc., to ensure that the receiving end can identify the sender's certificate and use the correct key for decryption. When the data transmission starts, the receiving end will first verify the certificate of the sending end (i.e., the collector deployed in various office networks or intranet environments). This step is completed through the TLS handshake, during which the signature, validity period, and trust relationship in the certificate chain of the certificate are checked. If the certificate is valid and trusted, the handshake is successful and the data transmission channel is established. After the TLS handshake is successful, the collector will send the encrypted data to the receiving end through the established secure channel. The encrypted data is transmitted under the protection of the TLS protocol, ensuring the security and integrity of the data during transmission. After receiving the encrypted data, the receiving end can use the decryption mechanism provided by the TLS protocol to decrypt the received data. The decryption process uses the session key negotiated during the handshake phase, which is generated based on the private and public keys of both parties through the Diffie-Hellman algorithm, ensuring the efficiency and security of data decryption. After decryption, the receiving end can also perform a preliminary verification of the data to check whether the data is complete and error-free. Once it is confirmed that the data is complete, the log data can be parsed and preprocessed to convert the data into a format that can be further analyzed.

[0051] Figure 3 is a schematic diagram of multi-location collection of the data processing method according to an optional embodiment of the present invention, as Figure 3As shown, network environments A1 and A2 represent the office or internal network environments of an enterprise located at different locations. Each environment has business applications and security systems, such as DLP (Data Leak Prevention system) and UBA (User Behavior Audit system). In each network environment, business applications and security systems (such as access control systems, DLP, and UBA) generate log and alarm information. This information includes records of employees' activities in specific applications, data transmission records, and potential security violation events. Fluentd collectors are deployed in each network environment and are responsible for collecting log and alarm information generated by local systems. The collectors are configured with different input plugins and can receive data from multiple data sources (such as files, networks, Syslog, etc.). The Fluentd collectors can send data to the data analysis system in network environment A0 using two-way TLS authentication and the Forward transmission method. This method not only provides data transmission security (through TLS encryption) but also ensures data integrity (through the verification mechanism of the TLS protocol). A data analysis system and Fluentd collection nodes are deployed in network environment A0. The collection nodes are responsible for receiving data from collectors in other network environments and passing it on to the data analysis system for real-time processing.

[0052] Through the above process, this schematic diagram shows how to collect, transmit, process, and store system logs and alarm data from multiple geographical locations in real-time to form a comprehensive, secure, and real-time data analysis system for monitoring and analyzing employees' behaviors in the enterprise internal network, improving the efficiency and accuracy of security operations. This architecture is particularly suitable for large enterprises or organizations with office locations distributed globally, which need to uniformly manage and analyze data from each location.

[0053] Through the above steps, it can be ensured that even in a complex multi-data center network environment, the secure transmission and reception of data, as well as the integrity and accuracy of the data, are provided, laying a solid foundation for subsequent data analysis and security policy implementation. The use of the TLS protocol is an important means to ensure data security in modern data transmission, and the decryption and data verification processes are key links to ensure that the data has not been tampered with or damaged when received.

[0054] As an optional embodiment, user data in the network system is parsed from the initial data, including: dividing the initial data into multiple types of data based on preset configuration tags; using respective parsing methods corresponding to multiple types to parse multiple types of data to obtain user data.

[0055] Optionally, parsing user data in the network system from the initial data involves decrypting the collected encrypted data and classifying it according to preset configuration tags. Specifically, the configuration tags can include uba_log: application platform operation log; dlp_log: DLP log, access_log: access control system log, uba_alert: application platform operation alert, dlp_alert: DLP alert. Different types of data can be sent to the data analysis platform by category through tags, or sent to the data analysis platform with tags for the platform to statistically identify and parse. For different types of data, the parsing methods for the corresponding user data may also be different, so corresponding methods need to be adopted for data parsing.

[0056] Through the above process, user data is parsed from the initial encrypted data, providing necessary information for subsequent security analysis and behavior auditing.

[0057] As an alternative embodiment, multiple types include access log type, application platform operation log type, application platform operation alert type, data loss prevention system log type, and data loss prevention system alert type. Among them, the data of the access log type includes user identification, user location, user MAC address, and log information. The data of the application platform operation log type includes the application platform name, user platform name, and corresponding log information. The data of the application platform operation alert type includes the application platform name, user platform name, and corresponding alert information. The data of the data loss prevention system log type includes the user MAC address and corresponding log information. The data of the data loss prevention system alert type includes the user MAC address and corresponding alert information.

[0058] Optionally, Table 1 is a data type and field matching table. As shown in Table 1, it includes multiple types and the data included in each type, that is, fields. Among them, the access log is the log of the access control system, the application platform operation logger is the operation log in the user behavior auditing system, the application platform operation alert is the alert in the user behavior auditing system, the DLP log is the log of the data loss prevention system, and the DLP alert is the alert of the data loss prevention system. Different types include different fields. Subsequently, these data need to be managed with user data, so as to better conduct user behavior analysis.

[0059]

[0060]

[0061] Table 1 Data Type and Field Matching Table

[0062] As an alternative embodiment, corresponding parsing methods for various types are adopted to parse data of various types respectively to obtain user data, including: when the target data is of the data leakage prevention system log type or the data leakage prevention system alert type, determining the target user MAC address in the target data; determining the first correspondence between the user MAC address and the user data based on the data corresponding to the access log; and selecting the user data corresponding to the target user MAC address as the target user data corresponding to the target data based on the first correspondence.

[0063] Optionally, since DLP logs and alerts only contain MAC addresses and cannot directly identify user identities and user locations. However, for DLP logs / alerts, the user&mac address relationship in the access log can be used to identify which user they belong to. Because after the user device connects to the office network system, it needs to pass through the access system authentication first to access the network environment normally. Generally speaking, when the DLP data arrives, there is already corresponding access login log information. Therefore, the data stream (macStream) of the relationship between mac and user and the relationship between user and location can be extracted from the parsed access_log stream. The dlp_log and dlp_alert streams are connected to the macStream, and the MapState state is used in the connection operator to store the mappings of mac->user and user->location. When calculating the dlp_log and dlp_alert data, the corresponding user and location are found through their mac fields to obtain the user data.

[0064] Specifically, when the target data is the log or alert generated by the DLP system, the main parsing objective is to determine the target user's MAC address therein. This usually involves using the parsing plug-in or custom filter of Fluentd to process the data and extract the MAC address field according to the format of the DLP log. The access log usually contains user login information, including user identification, the MAC address of the device, and the user's location. Therefore, the first correspondence between the user's MAC address and user data can be constructed, that is, the mappings of mac->user and user->location. This relationship can be stored in the state backend of the streaming computing framework (such as Flink) for use in subsequent stream joins. Use the stream join operator in Flink to associate the DLP log stream with the access log stream based on the MAC address. In the implementation of the operator, the MAC address in the DLP log can be associated with the specific user and location information by querying the status mappings of mac->user and user->location. After the stream join is completed, the DLP log data will carry additional user information (such as 4A account and location information) to form the target user data. These data can be further processed, such as formatted into a unified log or alert format, or directly written into the database to form a panoramic record of employees' security behaviors.

[0065] Through the above steps, the data in the DLP system can be associated with the user information in the access log in real time, which not only improves the richness of the data, but also provides a more comprehensive data perspective for subsequent security analysis, helping to more accurately identify and respond to data security incidents. This entire process demonstrates the powerful capabilities of streaming computing in real-time data analysis and processing, especially when dealing with multi-source heterogeneous data.

[0066] As an alternative embodiment, various types of corresponding parsing methods are adopted to parse various types of data respectively to obtain user data, including: in the case where the target data is of the application platform operation log type or the application platform operation alert type, extracting the target application platform name and the target application platform user name in the target data; obtaining the target user identification corresponding to the target application platform user name in the target application platform; determining the second correspondence between the user identification and the user location based on the data corresponding to the access log; selecting the user location corresponding to the target user identification as the target user location corresponding to the target data based on the second correspondence; and determining the target user data corresponding to the target data according to the target user identification and the target user location.

[0067] Optionally, since only app_user among the operation logs and alerts of the application platform does not have a 4A account and thus cannot directly identify employees, for the operation logs and alerts of the application platform, because the 4A system uniformly manages the account systems of all internal application systems, the relationship between user and app in the 4A system and the app_user (the user name of the application platform) can be loaded into the scenario engine as a broadcast stream at regular intervals, and this broadcast stream is connected to the operation logs and alert streams of the application platform. The MapState is used to store the app+user->user mapping in the connection operator, and the corresponding user is found through its app and user fields during the uba_log and uba_alert data calculation. After obtaining the user, it is connected to the macStream, and the corresponding location is obtained through the user->location mapping in the connection operator to obtain the user data.

[0068] Specifically, the name of the application platform (app) and the name of the user who performed the operation on this platform (app_user) can be extracted from the data record. This is usually achieved by configuring a Map operator in the Flink job, which can parse each record in the data stream and extract the required fields. Since the user name (app_user) of the application platform may not be consistent with the user identifier (user) in the 4A system, the correct user identifier needs to be obtained from the 4A system. This can be done by setting up a broadcast stream that contains the mapping relationship between the application platform user name and the 4A user name. In Flink, the BroadcastProcessFunction operator can receive the broadcast stream and associate it with the uba_log or uba_alert stream to find and replace app_user with the corresponding user identifier. The user data extracted from the access_log (admission log) stream can be used, which contains the user identifier (user), the device MAC address (mac), and the user's location (location). Through the stream connection operator, the mapping relationship between user and location is stored using the state backend to form the second corresponding relationship between the user identifier and the user's location. During the processing of the uba_log and uba_alert streams, once the target user identifier (user) is obtained, the location corresponding to this user identifier can be found by querying the second corresponding relationship stored in the state backend, that is, the user's location. Finally, using the extracted user identifier and location information, a data record containing complete user information is constructed, which is the target user data.

[0069] Through the above steps, data from different systems can be effectively parsed and associated to form a panoramic record of employees' safety behaviors, providing strong support for subsequent data security analysis. This processing method combines the real-time and high-efficiency characteristics of stream computing with the flexibility of state management to achieve real-time data association and processing.

[0070] As an alternative embodiment, the collector is a Fluentd collector.

[0071] Optionally, the Fluentd collector is suitable for complex environments with multi-systems and multi-source data. Since Fluentd supports multiple operating systems, including Linux, Windows, and macOS, this means it can run on different servers and devices to achieve unified data collection. And it can process various formats of log data, such as JSON, CSV, text, etc., and convert them into a unified format for transmission. This is particularly useful when dealing with heterogeneous data from different systems (such as access control systems, 4A, DLP, and UBA systems), simplifying the complexity of subsequent data processing.

[0072] In summary, Fluentd, as a log collector, plays a key role in the data collection and preprocessing stage. It can not only handle multi-source heterogeneous data but also ensure the secure transmission of data. At the same time, it provides important functions such as data format conversion, real-time transmission, error handling, and logging, making it an important tool for building an efficient, secure, and scalable data collection and transmission system.

[0073] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence because, according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0074] Through the description of the above embodiments, those skilled in the art can clearly understand that the data processing method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0075] According to an embodiment of the present invention, there is also provided a data processing apparatus for implementing the above data processing method. Figure 4 It is a structural block diagram of the data processing apparatus provided according to an embodiment of the present invention, as Figure 4 shown. The data processing apparatus includes: a receiving module 402, an analysis module 404, an association module 406, and a storage module 408. The data processing apparatus will be described below.

[0076] The receiving module 402 is configured to receive initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system.

[0077] The analysis module 404 is connected to the receiving module 402 and is configured to analyze user data in the network system from the initial data, where the user data includes a user identifier and a user's location.

[0078] The association module 406 is connected to the analysis module 404 and is configured to associate the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data.

[0079] The storage module 408 is connected to the association module 406 and is configured to store the target data in a preset database.

[0080] Optionally, the receiving module is configured to receive initial data, including: a receiving unit configured to receive encrypted data in the collector, where the encrypted data is data encrypted based on the TLS protocol; a verification unit configured to verify the certificate of the collector; and a decryption unit configured to decrypt the encrypted data based on the TLS protocol to obtain the initial data when the verification is passed.

[0081] Optionally, the analysis module is configured to analyze user data in the network system from the initial data, including: a partitioning unit configured to partition the initial data into multiple types of data based on a preset configuration tag; and an analysis unit configured to analyze the multiple types of data respectively using the corresponding analysis methods for each type of data to obtain the user data.

[0082] Optionally, there are multiple types including access log type, application platform operation log type, application platform operation alarm type, data leakage prevention system log type, and data leakage prevention system alarm type. Among them, the data of the access log type includes user identification, user location, user MAC address, and log information; the data of the application platform operation log type includes application platform name, user platform name, and corresponding log information; the data of the application platform operation alarm type includes application platform name, user platform name, and corresponding alarm information; the data of the data leakage prevention system log type includes user MAC address and corresponding log information; the data of the data leakage prevention system alarm type includes user MAC address and corresponding alarm information.

[0083] Optionally, use the respective parsing methods corresponding to multiple types to parse the data of multiple types respectively to obtain user data, including: a first determination unit, configured to determine the target user MAC address in the target data when the target data is of the data leakage prevention system log type or the data leakage prevention system alarm type; a second determination unit, configured to determine the first correspondence between the user MAC address and the user data based on the data corresponding to the access log; a first selection unit, configured to select the user data corresponding to the target user MAC address as the target user data corresponding to the target data based on the first correspondence.

[0084] Optionally, use the respective parsing methods corresponding to multiple types to parse the data of multiple types respectively to obtain user data, including: an extraction unit, configured to extract the target application platform name and the target application platform user name in the target data when the target data is of the application platform operation log type or the application platform operation alarm type; an acquisition unit, configured to acquire the target user identification corresponding to the target application platform user name in the target application platform; a third determination unit, configured to determine the second correspondence between the user identification and the user location based on the data corresponding to the access log; a second selection unit, configured to select the user location corresponding to the target user identification as the target user location corresponding to the target data based on the second correspondence; a fourth determination unit, configured to determine the target user data corresponding to the target data according to the target user identification and the target user location.

[0085] Optionally, the collector is a Fluentd collector.

[0086] It should be noted here that the above receiving module 402, parsing module 404, association module 406, and storage module 408 correspond to steps S202 to S208 in the embodiment. The instances and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in the embodiment.

[0087] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the above computer device may be at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0088] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely set relative to the processor, and these remote memories can be connected to the computer terminal through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and their combinations.

[0089] The processor can call the information and application programs stored in the memory through a transmission device to execute the following steps: receiving initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; parsing user data in the network system from the initial data, where the user data includes a user identifier and a user's territorial location; associating the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; and storing the target data in a preset database.

[0090] By adopting the embodiments of the present invention, a way of a data processing method is provided. By receiving initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; parsing user data in the network system from the initial data, where the user data includes a user identifier and a user's territorial location; associating the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; and storing the target data in a preset database, the purpose of summarizing and storing user data for facilitating user behavior analysis is achieved, thereby realizing the technical effects of improving the response speed to security threats and enhancing the security of the system. Furthermore, the technical problem that there may be multiple different systems in the current enterprise internal network environment, and the data stored between each system is independent of each other, making it difficult to quickly and accurately analyze user behavior, thus affecting the timely identification and response to security threats, is solved.

[0091] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and this program can be stored in a non-volatile storage medium. The storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disc, etc.

[0092] An embodiment of the present invention also provides a non-volatile storage medium. Optionally, in this embodiment, the above non-volatile storage medium can be used to store the program code executed by the data processing method provided in the above embodiment.

[0093] Optionally, in this embodiment, the above non-volatile storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0094] Optionally, in this embodiment, the non-volatile storage medium is set to store the program code for performing the following steps: receiving initial data, where the initial data is obtained by the collector collecting data from multiple subsystems in the network system; parsing the user data in the network system from the initial data, where the user data includes a user identifier and a user's place of residence; associating the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; and storing the target data in a preset database.

[0095] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can implement: receiving initial data, where the initial data is obtained by the collector collecting data from multiple subsystems in the network system; parsing the user data in the network system from the initial data, where the user data includes a user identifier and a user's place of residence; associating the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; and storing the target data in a preset database.

[0096] The above serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0097] In the above embodiments of the present invention, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0099] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0101] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0102] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, Including: Receiving initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; Parsing user data in the network system from the initial data, where the user data includes a user identifier and a user's territorial location; Associating the user data with other data to obtain target data, where the other data is data corresponding to the user data in the initial data; Storing the target data in a preset database.

2. The method according to claim 1, characterized in that, The receiving of the initial data includes: Receiving encrypted data in the collector, where the encrypted data is data encrypted based on the TLS protocol; Verifying the certificate of the collector; When the verification is passed, decrypting the encrypted data based on the TLS protocol to obtain the initial data.

3. The method according to claim 1, characterized in that, The parsing of the user data in the network system from the initial data includes: Dividing the initial data into multiple types of data based on a preset configuration tag; Using respective parsing methods corresponding to the multiple types to parse the multiple types of data respectively to obtain the user data.

4. The method according to claim 3, wherein The multiple types include an access log type, an application platform operation log type, an application platform operation alarm type, a data leakage prevention system log type, and a data leakage prevention system alarm type. Among them, the data of the access log type includes a user identifier, a user's territorial location, a user MAC address, and log information; the data of the application platform operation log type includes an application platform name, a user platform name, and corresponding log information; the data of the application platform operation alarm type includes an application platform name, a user platform name, and corresponding alarm information; the data of the data leakage prevention system log type includes a user MAC address and corresponding log information; the data of the data leakage prevention system alarm type includes a user MAC address and corresponding alarm information.

5. The method according to claim 4, characterized in that, The using of respective parsing methods corresponding to the multiple types to parse the multiple types of data respectively to obtain the user data includes: When the target data is of the data leakage prevention system log type or the data leakage prevention system alarm type, determining the target user MAC address in the target data; Based on the data corresponding to the access log, determining a first correspondence between the user MAC address and the user data; Based on the first correspondence, selecting the user data corresponding to the target user MAC address as the target user data corresponding to the target data.

6. The method according to claim 4, characterized in that, The using of respective parsing methods corresponding to the multiple types to parse the multiple types of data respectively to obtain the user data includes: When the target data is of the application platform operation log type or the application platform operation alarm type, extracting the target application platform name and the target application platform user name in the target data; Obtaining the target user identifier corresponding to the target application platform user name in the target application platform; Based on the data corresponding to the access log, determining a second correspondence between the user identifier and the user's territorial location; Based on the second corresponding relationship, select the user's place of residence corresponding to the target user identifier as the target user's place of residence corresponding to the target data; Determine the target user data corresponding to the target data according to the target user identifier and the target user's place of residence.

7. The method according to any one of claims 1 to 6, characterized in that, The collector is a Fluentd collector.

8. A data processing device, characterized in that, Comprising: A receiving module, configured to receive initial data, where the initial data is obtained by a collector collecting data from multiple subsystems in a network system; An analysis module, configured to analyze the user data in the network system from the initial data, where the user data includes a user identifier and a user's place of residence; An association module, configured to associate the user data with other data to obtain target data, where the other data is the data corresponding to the user data in the initial data; A storage module, configured to store the target data in a preset database.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, where when the program runs, it controls the device where the non-volatile storage medium is located to execute the data processing method according to any one of claims 1 to 7.

10. A computer device, characterized in that, Comprising: A memory and a processor, The memory stores a computer program; The processor is configured to execute the computer program stored in the memory, and when the computer program runs, it causes the processor to execute the data processing method according to any one of claims 1 to 7.

Images