Data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a data processing method and device, electronic equipment, computer readable storage medium and computer program product; the method comprises the following steps: in response to that a first gateway in a gateway cluster establishes a connection with a target client in a plurality of clients, sending a task command to the target client through the first gateway based on a mapping relationship between the first gateway and the target client; receiving target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; determining the data type of the target data; performing data processing on the target data according to the data processing mode corresponding to the data type. Through the application, the gateway cluster can be used as a buffer to temporarily store a large amount of target data generated during peak period, thereby relieving the data processing pressure of the server in a high concurrency scenario.

Description

Technical Field

[0001] This application relates to the field of distributed systems technology, and in particular to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] The host is the basic unit for program execution, and a host security system is a security solution specifically designed to protect critical enterprise information systems. Host security systems ensure the security and business continuity of enterprise hosts through functions such as real-time detection, intrusion detection, malware protection, data backup and recovery. As host security systems continue to evolve and business demands increase, they need to possess higher performance and scalability. However, the host security systems used in related technologies cannot handle scenarios with sudden surges in traffic; when the data volume increases, it may lead to sluggish system response or even crashes. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that enables a gateway cluster to act as a buffer to temporarily store large amounts of target data generated during peak periods, thereby alleviating the data processing pressure on servers in high-concurrency scenarios.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, the method comprising: responding to a first gateway in a gateway cluster establishing a connection with a target client among multiple clients; sending a task command to the target client through the first gateway based on the mapping relationship between the first gateway and the target client; receiving target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; determining the data type of the target data; and performing data processing on the target data according to the data processing method corresponding to the data type.

[0006] This application provides a data processing apparatus, comprising: a command sending module, configured to send a task command to a target client through the first gateway in a gateway cluster, in response to a first gateway in a gateway cluster establishing a connection with a target client among multiple clients, based on a mapping relationship between the first gateway and the target client; a data receiving module, configured to receive target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; a determining module, configured to determine the data type of the target data; and a data processing module, configured to process the target data according to a data processing method corresponding to the data type.

[0007] In the above scheme, the command sending module is further configured to: obtain the client identifier of the target client in response to the command message that generates the task command; determine the first gateway that establishes a connection with the target client from the database based on the mapping relationship between the gateway identifier of the first gateway and the client identifier of the target client; send the command message of the task command to the first gateway; and send the command message of the task command to the target client through the first gateway.

[0008] In the above scheme, the command sending module is further configured to, in response to the startup of a target client among the plurality of clients, determine the first gateway from the gateway cluster; send the first network parameters of the first gateway to the target client; receive the first connection request sent by the target client according to the first network parameters; the first connection request includes the gateway identifier of the first gateway; in response to the first connection request, control the target client to connect with the first gateway; and map the gateway identifier to the client identifier and store it in the database.

[0009] In the above scheme, the command sending module is further configured to: delete the mapping relationship between the gateway identifier and the client identifier from the database in response to the target client disconnecting from the first gateway; determine the second gateway from the gateway cluster in response to the target client re-establishing a connection after disconnecting from the first gateway; send the second network parameters of the second gateway to the target client; receive the second connection request sent by the target client according to the second network parameters; the second connection request includes the gateway identifier of the second gateway; control the target client to connect to the second gateway in response to the second connection request; and store the mapping between the gateway identifier of the second gateway and the client identifier in the database.

[0010] In the above scheme, the data processing module is further configured to: determine the target data category of the target data in response to the target data being of a first real-time type; obtain the target data from the message queue corresponding to the target data category; wherein the real-time indicator value of the target data having the first real-time type is less than a preset real-time indicator value threshold; determine the target data processing thread corresponding to the target data category; and call the target data processing thread to process the target data.

[0011] In the above scheme, the data processing module is further configured to, in response to the existence of a message queue corresponding to the target data category in the current message queue set, store the target data in the message queue corresponding to the target data category; and, in response to the absence of a message queue corresponding to the target data category in the current message queue set, create a message queue corresponding to the target data category in the message queue set and store the target data in the message queue corresponding to the target data category.

[0012] In the above scheme, the data processing module is further configured to parse the target data in the message queue according to a preset parsing rule to obtain parsed data; determine the target data storage unit in a preset distributed data storage; and store the parsed data in the target data storage unit.

[0013] In the above scheme, the device further includes a gateway upgrade module, which is used to call a timer to start timing in response to the gateway service of the first gateway being in an upgrade state; and to re-establish the connection between the target client and the first gateway in response to the timer timing being equal to a preset duration.

[0014] In the above scheme, the gateway upgrade module is further configured to upgrade the first gateway service of the first gateway in response to obtain the upgraded first gateway service, and realize the connection between the first gateway and the target client through the upgraded first gateway service; the first gateway service is the gateway service currently used to realize the connection between the first gateway and the target client; and the first gateway service before the upgrade is terminated.

[0015] In the above scheme, the gateway upgrade module is further configured to: create a second gateway service for the first gateway; upgrade all gateway services of the first gateway and the second gateway service to obtain upgraded all gateway services and upgraded second gateway services; for each upgraded gateway service among the upgraded all gateway services and the upgraded second gateway service, determine whether each upgraded gateway service has a client connection record; and delete any upgraded gateway service from the upgraded gateway services that do not have a client connection record.

[0016] This application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the data processing method provided in this application.

[0017] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0018] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0019] The embodiments of this application have the following beneficial effects:

[0020] When the first gateway in the gateway cluster establishes a connection with a target client among multiple clients, task commands can be sent to the target client through the first gateway based on the mapping relationship between the first gateway and the target client. The target client generates target data according to the task commands, and this target data can be stored by the first gateway. The gateway cluster can be used as a buffer to temporarily store large amounts of target data generated during peak periods. The data is then returned through the first gateway. By managing the connection between the server and client through the gateway cluster, the connection between the server and client is decoupled, allowing both the server and client to scale independently. Then, after determining the data type of the target data, data processing is performed according to the data processing method corresponding to the data type. In this way, not only can the decoupling of the connection between the server and client be achieved by introducing the gateway cluster to manage the connection between the server and client, allowing both the server and client to scale independently, but the gateway cluster can also act as a buffer to temporarily store large amounts of target data generated during peak periods, thereby alleviating the data processing pressure on the server in high-concurrency scenarios. Attached Figure Description

[0021] Figure 1 This is an optional architecture diagram of the data processing system provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0023] Figure 3 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0024] Figure 4This is another optional flowchart illustrating the data processing method provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram illustrating the command sending implementation process provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of the data processing implementation flow provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the architecture of the host security system provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the server horizontal scaling architecture provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the client access process provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram illustrating an implementation of data storage provided in an embodiment of this application;

[0031] Figure 11 This is another schematic diagram illustrating the data storage implementation provided in the embodiments of this application;

[0032] Figure 12 This is another schematic diagram of the host security system provided in the embodiments of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0038] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0039] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0040] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0041] 2) Human-computer interaction interface, which is used to provide human-computer interaction functions / display data.

[0042] For example, graphical user interfaces (GUIs) include augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing users to see stereoscopic images without wearing special glasses), multimodal interfaces (interfaces that combine multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0043] 3) Cloud computing: A network that provides resources, including computing and storage capabilities.

[0044] 4) Container orchestration platform (Kubernetes, K8S): An open-source container orchestration platform for running distributed applications and services at scale.

[0045] 5) Rolling update: This refers to updating a portion of the containers (Pods) at a time, rather than shutting down all Pods on the server at the same time and then updating them (such as a replacement-force solution). Updating one by one can avoid business interruption.

[0046] 6) Cloud Workload Protection (CWP) System: This system provides users with security protection services such as asset management, Trojan file detection and removal, hacker intrusion prevention, vulnerability risk warning and security baseline, helping enterprises build a server security protection system.

[0047] 7) Naming Service: A crucial service in distributed and microservice architectures, used to solve service registration and discovery, dynamic routing, load balancing and fault tolerance issues in remote calls.

[0048] 8) Random weighting: The naming service randomly assigns an Internet Protocol (IP) address of a server to the client based on the weights set for each service.

[0049] 9) Gateway service: Used to establish connections with clients, respond to client requests and reports, and has the ability to send commands to specified clients, but does not handle specific business logic.

[0050] 10) Backend services: Processing user business logic, such as risk identification and data archiving.

[0051] 11) Message Queue: A container for storing messages. When a message is needed, it is retrieved directly from the container. Because a queue is a first-in, first-out (FIFO) data structure, messages are consumed in order. For example, target data can be stored as messages in a message queue, and the first target data to enter will be the first to leave.

[0052] 12) Topic: In a message queue, a Topic is a logical concept used to organize and categorize messages. A Topic is the basic unit in a message queue, used to logically group and classify messages. For example, one Topic corresponds to each data category.

[0053] Host security systems in related technologies typically involve installing a client on the user's host to collect and report data, while the backend stores and analyzes the data on a server. However, when faced with sudden surges in traffic, these host security systems cannot handle the massive amounts of data, potentially leading to sluggish responses or even crashes. They struggle to handle large-scale traffic and fail to meet the security needs of enterprises.

[0054] To address at least one of the aforementioned problems with methods in related technologies, embodiments of this application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. These methods enable a gateway cluster to act as a buffer, temporarily storing large amounts of target data generated during peak periods, thereby alleviating the data processing pressure on servers in high-concurrency scenarios. When a first gateway in the gateway cluster establishes a connection with a target client among multiple clients, a task command can be sent to the target client through the first gateway based on the mapping relationship between the first gateway and the target client. The target client generates target data according to the task command, and this target data can be stored by the first gateway. The gateway cluster can act as a buffer, temporarily storing large amounts of target data generated during peak periods. The data is then returned through the first gateway. By managing the connection between the server and client through the gateway cluster, the connection between the server and client is decoupled, allowing the server and client to be independently expanded and utilized. Then, after determining the data type of the target data, the target data is processed according to the data processing method corresponding to the data type. In this way, not only can the connection between the server and the client be decoupled by introducing a gateway cluster to manage the connection between the server and the client, allowing the server and the client to scale independently, but the gateway cluster can also be used as a buffer to temporarily store a large amount of target data generated during peak periods, thereby alleviating the data processing pressure on the server in high-concurrency scenarios.

[0055] Here, we first describe an exemplary application of the data processing device in the embodiments of this application. This data processing device is an electronic device used to implement a data processing method. In one implementation, the data processing device (i.e., electronic device) provided in the embodiments of this application can be implemented as a terminal or as a server. In one implementation, the data processing device provided in the embodiments of this application can be implemented as any terminal with data processing capabilities, such as a laptop, tablet, desktop computer, mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, intelligent robot, smart home appliance, and smart vehicle device. In another implementation, the data processing device provided in the embodiments of this application can also be implemented as a server, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of this application. The following describes an exemplary application when the data processing device is implemented as a server.

[0056] See Figure 1 , Figure 1 This is an optional architecture diagram of the data processing system provided in this application embodiment. The data processing system 10 in this application embodiment includes at least a terminal 100-1, a terminal 100-2, a network 200, and a server 300. A data processing application is installed on the terminal 100-1, which can provide data processing functions. A target client is deployed on the terminal 100-2.

[0057] The following description uses the installation of a data processing application on a terminal as an example. In this embodiment, server 300 can be a server for the data processing application. Server 300 can constitute the data processing device of this embodiment, that is, the data processing method of this embodiment is implemented through server 300. Terminals 100-1 and 100-2 are connected to server 300 through network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0058] See Figure 1When a user wants to process data, they can input a data processing operation on the client side of the data processing application. This operation allows them to select the task corresponding to the data to be processed or input the task identifier of the task. Terminal 100-1 then encapsulates the task identifier into a task processing request and sends it to server 300 via network 200. In response to this request, when the first gateway in the gateway cluster establishes a connection with the target client among multiple clients, server 300 sends a task command to the target client through the first gateway, based on the mapping relationship between the first gateway and the target client. Then, server 300 receives the target data sent by the first gateway. Next, server 300 determines the data type of the target data. Finally, server 300 processes the target data according to the data processing method corresponding to the data type. After obtaining the data processing result, server 300 sends the result as the task processing result to terminal 100-1, which can then display the result on the current interface of the data processing application.

[0059] In some embodiments, the implementation steps of the data processing method can also be executed by terminal 100-1. That is, after determining the task corresponding to the data to be processed, terminal 100-1 establishes a connection between the first gateway in the gateway cluster and the target client among multiple clients in response to the first gateway establishing a connection. Based on the mapping relationship between the first gateway and the target client, terminal 100-1 sends a task command to the target client through the first gateway. Then, terminal 100-1 receives the target data sent by the first gateway. After that, terminal 100-1 determines the data type of the target data. Finally, terminal 100-1 processes the target data according to the data processing method corresponding to the data type.

[0060] The data processing method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The cloud server responds to the first gateway in the gateway cluster and establishes a connection with the target client among multiple clients. Based on the mapping relationship between the first gateway and the target client, the first gateway sends task commands to the target client. Alternatively, the cloud server can receive target data sent by the first gateway, determine the data type of the target data, or process the target data according to the data processing method corresponding to the data type.

[0061] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2The illustrated electronic device 400 may be a data processing device, which includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0062] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0064] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0065] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0066] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0067] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 452 for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; presentation module 453 for enabling the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., display screen, speaker, etc.); input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432.

[0068] In some embodiments, the data processing apparatus provided in this application can be implemented in software. Figure 2 A data processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a command sending module 4551, a data receiving module 4552, a determination module 4553, and a data processing module 4554. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0069] In other embodiments, the data processing apparatus provided in this application can be implemented in hardware. As an example, the data processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0070] Figure 3 This is an optional flowchart illustrating the data processing method provided in the embodiments of this application. The following will be combined with... Figure 3 The steps shown are explained as follows: Figure 3 As shown, taking the server as the execution subject of the data processing method as an example, the method includes the following steps S101 to S104:

[0071] Step S101: In response to the establishment of a connection between the first gateway in the gateway cluster and the target client among multiple clients, a task command is sent to the target client through the first gateway based on the mapping relationship between the first gateway and the target client.

[0072] In this embodiment, a gateway (Api_gateway) is a device or software in a network that can act as a communication interface between different networks or subnets. The main functions of a gateway include protocol conversion, data forwarding, security control, and load balancing. A gateway cluster refers to a distributed system composed of multiple gateways that work collaboratively to manage communication and process data between clients and servers. A gateway cluster can evenly distribute client requests across multiple gateways, avoiding overload on individual gateways and improving overall system performance. Through redundancy design, when one gateway fails, other gateways can take over its work, ensuring continuous system operation. During periods of high traffic or data peaks, a gateway cluster can act as a buffer, temporarily storing data to prevent server overload. A gateway cluster can also dynamically increase or decrease the number of gateways based on system load to adapt to different traffic demands.

[0073] The gateway cluster includes multiple gateways. The first gateway can be any gateway in the gateway cluster. When the first gateway establishes a connection with the target client among multiple clients, the first gateway can send task commands to the target client based on the mapping relationship between the first gateway and the target client.

[0074] For example, in a smart home system, a gateway cluster can manage various smart devices (clients). A connection can be established between the first gateway in the cluster and a smart thermostat (target client). Based on a pre-defined mapping (e.g., the mapping between device identifiers and gateway identifiers), the server can send a task command, such as "get current temperature data," to the smart thermostat through the first gateway. Similarly, a host security system is a client-server (C / S) architecture system. By installing a client (Agent) into the user's host environment, basic data detection and reporting can be performed. The first gateway in the cluster can establish a connection with an Agent (target client) on any host. The server can send a task command, such as "get current host network configuration parameters," to the Agent through the first gateway based on a pre-defined mapping.

[0075] Step S102: Receive target data sent by the first gateway.

[0076] Here, the target data is the data generated by the target client in response to the task command, and the target data is stored by the first gateway.

[0077] In this embodiment, the server receives target data sent by the first gateway. This target data is generated by the target client executing a task command after receiving it. After receiving the command message from the first gateway, the target client parses the message to obtain the specific task to be executed. The target client then executes the task corresponding to the command, generating the target data. The target client sends this target data to the first gateway, which can then either send the target data to the server or store it first and then send it to the server based on its load.

[0078] For example, after receiving a task command to "obtain current temperature data," a smart thermostat measures the current ambient temperature and sends the temperature data (target data) back to the first gateway. Upon receiving the temperature data, the first gateway can store it in its local database or cache for later transmission to the server. Similarly, in a host security system, the command message sent to the first gateway could instruct the target client to perform a security scan of the host, such as checking for system vulnerabilities or detecting malware. Upon receiving the command message, the target client parses its content to obtain the specific task instructions and executes the security scan accordingly. After completing the scan, the target client can encapsulate the scan results into a data packet and send it back to the first gateway, which then forwards it to the server.

[0079] Step S103: Determine the data type of the target data.

[0080] In this embodiment, after receiving the target data, the server can analyze the target data to determine its data type. For example, in a smart home system, the temperature data received by the first gateway is a numerical temperature value (e.g., 25.5 degrees Celsius). The first gateway receives target data returned by the smart lock in response to the task command "Query lock status," and determines that the returned target data can be of a Boolean type (e.g., "Yes" indicates the lock is locked, "No" indicates the lock is unlocked). The first gateway receives target data returned by the smart camera in response to the task command "Acquire real-time video stream," and determines that the returned target data can be of a multimedia data type (e.g., video stream).

[0081] Step S104: Perform data processing on the target data according to the data processing method corresponding to the data type.

[0082] In this embodiment, corresponding data processing methods can be pre-defined for different data types. Once the server determines the data type of the target data, it can process the target data according to the data processing method corresponding to the data type. The data processing method corresponding to each data type can be limited according to the actual situation, and this embodiment does not impose such limitations.

[0083] For example, once it's determined that the received temperature data is a temperature data type, the temperature data can be processed as follows: First, perform data validation, such as checking if the temperature value is within a reasonable range (e.g., whether the indoor temperature is between -10 and 40 degrees Celsius). Alternatively, perform data conversion, such as converting Celsius to Fahrenheit. Another option is data storage, storing the processed data in a more persistent storage system, such as a cloud database. Finally, data notification can be provided; if the temperature exceeds a set range (e.g., above 30 degrees Celsius), an alert notification can be sent to the user.

[0084] In this embodiment, when a first gateway in a gateway cluster establishes a connection with a target client among multiple clients, a task command can be sent to the target client through the first gateway based on the mapping relationship between the first gateway and the target client. The target client generates target data according to the task command, and the target data can be stored by the first gateway. The gateway cluster can be used as a buffer to temporarily store large amounts of target data generated during peak periods. The data is then returned through the first gateway. By managing the connection between the server and the client through the gateway cluster, the connection between the server and the client is decoupled, allowing the server and the client to scale independently. Then, after determining the data type of the target data, data processing is performed on the target data according to the data processing method corresponding to the data type. Thus, not only is the connection between the server and the client decoupled by introducing the gateway cluster to manage the connection, allowing the server and the client to scale independently, but the gateway cluster can also be used as a buffer to temporarily store large amounts of target data generated during peak periods, thereby alleviating the data processing pressure on the server in high-concurrency scenarios.

[0085] The following examples illustrate the application scenarios of the data processing method provided in the embodiments of this application.

[0086] In a smart home system, a server can manage various smart devices through a gateway cluster. The server receives data returned by these smart devices through the gateway, processes the data according to the corresponding processing methods, and then displays the results on the user interface or performs further analysis, such as determining whether automatic temperature adjustment is needed. This enables intelligent temperature control for smart homes, improving user comfort and energy efficiency.

[0087] In enterprise server management scenarios, the CWP system provides comprehensive security protection services. For example, when enterprise servers face risks of hacker intrusion and Trojan attacks, the CWP system client is installed in the enterprise's host environment. The CWP system can manage various clients through a gateway cluster. It also utilizes the asset management function within the client to automatically inventory asset information such as hosts, processes, and ports on the server. Based on data reported by the client, when the CWP system detects hacker intrusion, it will issue real-time alerts and provide handling suggestions to ensure enterprise data security and stable business operations.

[0088] The data processing method of this application embodiment will be described below in conjunction with the above scenario.

[0089] Figure 4 This is another optional flowchart illustrating the data processing method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps S201 to S208:

[0090] Step S201: The terminal receives the task processing operation input by the user.

[0091] Task processing operations include selection operations or input operations. The selection operation is used to select the task corresponding to the data to be processed, or the input operation is used to input the task identifier of the task corresponding to the data to be processed.

[0092] In step S202, the terminal encapsulates the task identifier into the task processing request.

[0093] A task processing request is used to request the server to process the task to be processed.

[0094] Step S203: The terminal sends a task processing request to the server.

[0095] In this embodiment of the application, the terminal sends a task processing request to the server to request the server to process the task to be processed.

[0096] In step S204, the server responds to the establishment of a connection between the first gateway in the gateway cluster and the target client among multiple clients, and sends a task command to the target client through the first gateway based on the mapping relationship between the first gateway and the target client.

[0097] In some embodiments, see Figure 5 , Figure 5 The step S204, "sending a task command to the target client through the first gateway based on the mapping relationship between the first gateway and the target client," can be achieved through the following steps S2041 to S2044:

[0098] Step S2041: In response to the command message that generates the task command, obtain the client identifier of the target client.

[0099] In this embodiment, when the server receives a task processing request from the terminal, it can generate a command message corresponding to the task command. A command message is a formatted message used to transmit instructions and data between the server, the security management system, and the terminal device. The command message may contain key information such as the command type, the target client's client identifier, and parameters, used to perform a specific operation. When the command message is generated, the target client's client identifier can be obtained from the task processing request. The client identifier can be text, a number, a combination of letters, a combination of special symbols, or a mixture of these elements (such as "Agent-001"). Each client identifier is used to uniquely identify a client.

[0100] For example, in a host security system, when a server receives a task processing request to detect a host, it can generate a corresponding command message and obtain the client identifier of the target client on that host from the task processing request.

[0101] In some embodiments, the following processes may be performed before step S2041: in response to the startup of a target client among multiple clients, a first gateway is determined from the gateway cluster; the first network parameters of the first gateway are sent to the target client; a first connection request is received from the target client based on the first network parameters; the first connection request includes the gateway identifier of the first gateway; in response to the first connection request, the target client is controlled to connect to the first gateway; the gateway identifier and the client identifier are mapped and stored in the database.

[0102] In this embodiment, when a target client among multiple clients starts up, the server can determine a first gateway from the gateway cluster. This first gateway can be used to establish a connection with the target client for communication. Then, the server can send the first network parameters of the first gateway to the target client. Upon receiving the first network parameters, the target client can send a first connection request to the first gateway based on these parameters. This first connection request may include the gateway identifier of the first gateway. When the server receives the first connection request, it can control the connection between the target client and the first gateway. After the target client connects to the first gateway, the server can map the gateway identifier to the client identifier to obtain a mapping relationship, which is then stored in a database.

[0103] The following example illustrates that a host security system may include multiple gateways (GW-001, GW-002, etc.). When a target client starts up and prepares to access the host security system, it can first establish a connection with one of the gateways in the gateway cluster. The host security system can select a gateway (i.e., the first gateway) from the gateway cluster to connect to the target client. For example, it can select a gateway as the first gateway to connect to the target client based on the load or coverage of each gateway. Alternatively, a naming service can be used, setting the naming service to a weighted random allocation algorithm to assign a gateway to the target client as the first gateway for connection. It can also randomly assign a gateway to the target client as the first gateway for connection. It should be noted that the specific method of gateway selection can be set according to the actual situation, and this embodiment does not limit this. Afterwards, the first network parameters of the first gateway (such as IP address, port number, etc.) can be sent to the target client so that the client can connect to the gateway. For example, the first network parameters of the first gateway GW-001 include IP address (192.168.1.100), port number (8080), and gateway identifier (GW-001). The target client sends a connection request to the first gateway using the received first network parameters. The connection request may include the gateway identifier of the first gateway. Upon receiving the connection request, the first gateway verifies its validity. If the verification is successful, a connection can be established with the target client; this connection can be a Transmission Control Protocol (TCP) long-lived connection. For subsequent management and communication needs, the gateway identifier and client identifier can be mapped to obtain a mapping relationship, which is then stored in a database.

[0104] Through the above processing method, the client can establish a connection with the gateway, and at the same time record the mapping relationship between the client and the gateway. By introducing a gateway cluster to manage the connection between the server and the client, the connection between the server and the client is decoupled, allowing the server and the client to be scaled independently.

[0105] Step S2042: Based on the mapping relationship between the gateway identifier of the first gateway and the client identifier of the target client, determine the first gateway that establishes a connection with the target client from the database.

[0106] In this embodiment of the application, the database can be used to store the mapping relationship between the gateway identifier and the client identifier. After the server obtains the client identifier of the target client, it can search the database for the gateway identifier that has a mapping relationship with the client identifier. After determining the gateway identifier, the first gateway to establish a connection with the target client can be determined.

[0107] Step S2043: Send a command message containing the task command to the first gateway.

[0108] In this embodiment of the application, after the server determines the first gateway that establishes a connection with the target client, it sends a command message of the task command to the first gateway.

[0109] Step S2044: Send a command message containing the task command to the target client through the first gateway.

[0110] In this embodiment, the server sends a task command message to the target client through a first gateway. Upon receiving the task command message, the first gateway can forward the task command to the target client according to the client list managed by the first gateway. For example, the first gateway establishes connections with three clients (Agent-001, Agent-002, and Agent-003), with the target client represented as Agent-001. After receiving the task command message from the server, the first gateway identifies the target client as Agent-001 and then forwards the command message to Agent-001.

[0111] Through steps S2041 to S2044, collaborative work between the server, gateway, and client can be achieved, and the connection between the server and client can be managed through the gateway. This decouples the connection between the server and client, allowing the server and client to expand independently. Furthermore, through pre-stored mapping relationships, the server can quickly determine the first gateway connected to the target client, reducing latency in the task allocation process, improving the efficiency of task command transmission, and thus improving the overall efficiency of task execution.

[0112] Step S205: The server receives the target data sent by the first gateway.

[0113] Here, the target data is the data generated by the target client in response to the task command, and the target data is stored by the first gateway.

[0114] Step 206: The server determines the data type of the target data.

[0115] It should be noted that steps S205 to S206 are the same as steps S102 to S103 above. The implementation details of steps S205 to S206 in this embodiment will not be repeated.

[0116] In step S207, the server processes the target data according to the data processing method corresponding to the data type.

[0117] In some embodiments, see Figure 6 , Figure 6 Step S207 can be achieved through the following steps S2071 to S2074:

[0118] Step S2071: In response to the target data being of the first real-time type, determine the target data category of the target data.

[0119] In this embodiment, when the data type of the target data is a first real-time type, the server determines the target data category. The data type can be preset; for example, the data type may include a first real-time type and a second real-time type. Target data with a real-time indicator value less than a preset real-time indicator value threshold can be set as data of the first real-time type, and target data with a real-time indicator value greater than or equal to the preset real-time indicator value threshold can be set as data of the second real-time type. It should be noted that the data type and real-time indicator value threshold can be set according to actual conditions, and this embodiment does not limit this. For example, in a host security system, target data may include intrusion detection data, system performance data, security alarm data, real-time log data, session management data, configuration management data, historical log data, backup data, statistical report data, and software update data, etc. A real-time indicator value threshold can be determined based on the real-time requirements of these data. Data with a real-time indicator value less than the preset real-time indicator value threshold is designated as data of the first real-time type, and data with a real-time indicator value greater than or equal to the preset real-time indicator value threshold is designated as data of the second real-time type. For example, configuration management data is the host's configuration information and management data. Configuration management data typically doesn't require real-time updates and can be synchronized periodically, thus classifying it as a primary real-time data type. For example, historical log data consists of system and application logs from a past period. Primarily used for post-incident analysis and auditing, historical log data doesn't require real-time processing, making it a primary real-time data type as well. Similarly, intrusion detection data is used to detect network traffic and system behavior in real-time, identifying potential intrusions and requiring rapid detection and response to prevent further attack spread; therefore, intrusion detection data can be classified as a secondary real-time data type. Data belonging to the primary real-time data type can also have different data categories. For instance, both configuration management data and historical log data are primary real-time data, but their formats differ; in this case, configuration management data and historical log data can each be considered a separate data category.

[0120] Step S2072: Obtain the target data from the message queue corresponding to the target data category.

[0121] Here, the real-time index value of the target data with the first real-time type is less than the preset real-time index value threshold.

[0122] In this embodiment, a message queue is a middleware for transmitting messages between different applications or components, essentially acting as a container for storing messages. A message is a data unit transmitted in a message queue, typically a sequence of bytes. Messages can contain various types of data, such as text or binary data. Messages can also contain metadata (such as timestamps, priorities, etc.) and a message body (the actual data content). The queue is a message storage container used to temporarily store messages. A queue can be viewed as a first-in, first-out (FIFO) data structure, where messages enter the queue sequentially and are consumed sequentially. Different data categories correspond to different message queues, and the server can retrieve target data from the message queue corresponding to the target data category.

[0123] In some embodiments, the following processes may be performed before step S2072: in response to the existence of a message queue corresponding to the target data category in the current message queue set, the target data is stored in the message queue corresponding to the target data category; in response to the absence of a message queue corresponding to the target data category in the current message queue set, a message queue corresponding to the target data category is created in the message queue set, and the target data is stored in the message queue corresponding to the target data category.

[0124] In this embodiment, when the server receives data belonging to the first real-time type, it can save the data to a message queue for temporary storage, and then obtain the corresponding data by consuming messages in the message queue. When saving data, if a message queue corresponding to the target data type exists in the current message queue set, the target data can be stored in the message queue corresponding to the target data type. If a message queue corresponding to the target data type does not exist in the current message queue set, a message queue corresponding to the target data type is created in the message queue set, and the target data is stored in the message queue corresponding to the target data type.

[0125] For example, if the target data belongs to the category of configuration management data, and a corresponding message queue exists in the current message queue set, the configuration management data can be stored in that message queue. If the target data belongs to the category of historical log data, but a corresponding message queue does not exist in the current message queue set, a corresponding message queue can be created in the message queue set, and then the historical log data can be stored in the newly created message queue.

[0126] Message queues can temporarily store excess target data during peak traffic periods, reducing the impact of excessive target data on the server and allowing the server to process the excess target data gradually, thereby achieving reasonable resource utilization.

[0127] In some embodiments, after storing the target data in the message queue corresponding to the target data category, the following processing may also be performed: parsing the target data in the message queue according to a preset parsing rule to obtain parsed data; determining the target data storage unit in a preset distributed data storage; and storing the parsed data in the target data storage unit.

[0128] In this embodiment, parsing rules and a distributed data storage can be pre-defined. The parsing rules can be used to convert the original target data into a specified format. Parsing rules may include data format conversion, field extraction, data cleaning, etc., which are not limited in this embodiment. The distributed data storage can have multiple data storage units. After storing the target data in the message queue corresponding to the target data category, the server can parse the target data in the message queue according to the pre-defined parsing rules to obtain parsed data. Furthermore, the target data storage unit is determined within the pre-defined distributed data storage. This can be done by randomly selecting a data storage unit or by selecting the data storage unit with the largest storage space. The specific selection method can be set according to the actual situation, and is not limited in this embodiment.

[0129] Through the above processing, the target data in the message queue can be parsed according to the preset parsing rules to obtain parsed data. Then, the target data storage unit is determined in the preset distributed data storage and the parsed data is stored in the target data storage unit, which ensures the standardization of data and the efficiency of storage. It is suitable for processing a large amount of different types of data and improves the server's ability to process large amounts of data.

[0130] Step S2073: Determine the target data processing thread corresponding to the target data category.

[0131] In this embodiment, different data categories correspond to different data processing threads. The data processing thread for each data category can be set according to actual conditions, and this embodiment does not impose any limitations on this. Once the server determines the target data category of the target data, it can determine the target data processing thread corresponding to that category. For example, suppose a thread specifically for processing configuration management data is pre-defined and named "Config Management Thread". The server can find the corresponding processing thread based on the target data category "Configuration Management Data".

[0132] Step S2074: Invoke the target data processing thread to process the target data.

[0133] In this embodiment, the server invokes a target data processing thread to process the target data. For example, the target data processing thread is a "Config Management Thread," and the server can invoke the "ConfigManagement Thread" to perform specific operations, such as updating the server's network configuration file.

[0134] Through steps S2071 to S2074, the target data is classified according to its real-time type, and the target data is temporarily stored using a message queue. This allows the target data to be processed step by step, effectively alleviating server pressure during peak traffic periods and enabling flexible management of data with different real-time requirements. At the same time, by allocating dedicated processing threads to different data categories, the efficiency and accuracy of data processing are ensured.

[0135] In some embodiments, step S207 may also be implemented in the following manner: in response to the data type of the target data being a second real-time type, determining the target data category of the target data; wherein the real-time index value of the target data having the second real-time type is greater than or equal to a preset real-time index value threshold; determining the target data processing thread corresponding to the target data category; and calling the target data processing thread to process the target data.

[0136] In this embodiment, when the data type of the target data is a second real-time type, the target data category is determined. Then, the target data processing thread corresponding to the target data category is determined, and the target data processing thread is invoked to process the target data. For example, intrusion detection data is used to detect network traffic and system behavior in real time, detecting potential intrusion behavior. It requires rapid detection and response to prevent further spread of attacks, and can therefore be classified as second real-time type data. Intrusion detection data can be treated as a data category, and a dedicated thread for processing intrusion detection data, named "Security Thread," can be pre-defined. The server can find the corresponding processing thread based on the data category "Intrusion Detection Data" and utilize the "Security Thread" for data processing.

[0137] By classifying the target data according to the real-time type in the above manner, data with high real-time requirements can be processed according to the data processing thread corresponding to the data category, so as to process the data in a timely manner and improve the data processing efficiency.

[0138] In step S208, the server sends the task processing result to the terminal.

[0139] Through the application embodiments, collaborative work between servers, gateways, and clients can be achieved. The gateway manages the connection between the server and client, decoupling their relationship and allowing them to scale independently. Furthermore, through pre-stored mapping relationships, the server can quickly determine the first gateway connected to the target client, reducing latency in task allocation, improving task command transmission efficiency, and thus enhancing overall task execution efficiency. The target client generates target data based on task commands. This target data can be stored by the first gateway, using the gateway cluster as a buffer to temporarily store large amounts of target data generated during peak periods. The data is then returned through the first gateway. The gateway cluster manages the connection between the server and client, decoupling their relationship and allowing for independent scaling and utilization. The target data is then categorized based on real-time performance metrics and processed according to the corresponding data processing methods. This approach not only decouples the server and client by introducing a gateway cluster to manage their connection, enabling independent scaling, but also uses the gateway cluster as a buffer to temporarily store large amounts of target data generated during peak periods, thereby alleviating the data processing pressure on the server in high-concurrency scenarios.

[0140] In some embodiments, when the target client disconnects from the first gateway, the following processes may also be performed: in response to the target client disconnecting from the first gateway, the mapping relationship between the gateway identifier and the client identifier is deleted from the database; in response to the target client re-establishing a connection after disconnecting from the first gateway, a second gateway is determined from the gateway cluster; the second network parameters of the second gateway are sent to the target client; a second connection request is received from the target client based on the second network parameters; the second connection request includes the gateway identifier of the second gateway; in response to the second connection request, the connection between the target client and the second gateway is controlled; the gateway identifier of the second gateway is mapped to the client identifier and stored in the database.

[0141] In this embodiment, during practical application, the target client may disconnect from the first gateway due to various reasons (such as the target client being restarted). In this case, the mapping relationship between the target client's client identifier and the first gateway's gateway identifier stored in the database can be deleted. Afterwards, when the target client disconnects from the first gateway, it can re-establish a connection with the gateway. However, the gateway with which it re-establishes the connection can be the original first gateway or a second gateway in the gateway cluster that is different from the first gateway. It should be noted that the implementation of "determining the second gateway from the gateway cluster; sending the second network parameters of the second gateway to the target client; receiving the second connection request sent by the target client based on the second network parameters; the second connection request including the gateway identifier of the second gateway; responding to the second connection request, controlling the connection between the target client and the second gateway; and mapping the gateway identifier of the second gateway to the client identifier and storing it in the database" can be referenced from the implementation of "determining the first gateway from the gateway cluster; sending the first network parameters of the first gateway to the target client; receiving the first connection request sent by the target client based on the first network parameters; the first connection request including the gateway identifier of the first gateway; responding to the first connection request, controlling the connection between the target client and the first gateway; and mapping the gateway identifier to the client identifier and storing it in the database". This will not be elaborated upon here.

[0142] By performing the above processing, the original mapping relationship can be deleted when the target client disconnects from the first gateway, ensuring that only the connection information between the client and the gateway is retained in the database, thus reducing the probability of server command failure.

[0143] In some embodiments, when the gateway service of the first gateway is upgraded, the following processes may also be performed: in response to the gateway service of the first gateway being in an upgrade state, a timer is invoked to start timing; in response to the timer timing being equal to a preset duration, the connection between the target client and the first gateway is re-established.

[0144] In this embodiment, the gateway service is software or an application running on the first gateway device, responsible for implementing various functions of the first gateway. The gateway service can be a background process or service providing data processing, forwarding, security, and other functions. When the gateway service of the first gateway is in an upgrade state, the target client disconnects from the first gateway. At this time, a timer can be invoked to time the process, with a preset waiting duration, which can be set to be longer than the time required for the gateway service upgrade. When the timer reaches the preset duration, the connection between the target client and the first gateway is re-established. For example, if the gateway service upgrade requires 1 minute, the preset duration can be 2 minutes; when the timer reaches 2 minutes, the connection between the target client and the first gateway is re-established.

[0145] The above process allows a timer to be invoked and a preset duration to be waited for, enabling the client to re-establish the connection after the gateway service upgrade is complete. This ensures stability and reliability during the gateway upgrade and avoids connection interruptions caused by gateway service unavailability.

[0146] In some embodiments, when the gateway service of the first gateway is upgraded, the following processes may also be performed: in response to upgrading the first gateway service of the first gateway, an upgraded first gateway service is obtained, and the connection between the first gateway and the target client is realized through the upgraded first gateway service; the first gateway service is the gateway service currently used to realize the connection between the first gateway and the target client; the first gateway service before the upgrade is terminated.

[0147] In this embodiment, the first gateway service is the gateway service currently used to establish a connection between the first gateway and the target client. When the first gateway service of the first gateway is upgraded to obtain an upgraded first gateway service, the upgraded first gateway service can be used to establish a connection between the first gateway and the target client. Once the connection between the first gateway and the target client has been established through the upgraded first gateway service, the original first gateway service can be terminated.

[0148] By implementing the above steps—upgrading the first gateway service and re-establishing the connection with the target client while terminating the old version of the service—a stable service for the client can be ensured.

[0149] In some embodiments, when the gateway service of the first gateway is upgraded, the following processes may also be performed: creating a second gateway service for the first gateway; upgrading all gateway services of the first gateway and the second gateway service to obtain upgraded all gateway services and upgraded second gateway services; for each upgraded gateway service in the upgraded all gateway services and upgraded second gateway services, determining whether each upgraded gateway service has a client connection record; deleting any upgraded gateway service from the upgraded gateway services that do not have a client connection record.

[0150] In this embodiment, firstly, a second gateway service is created for the first gateway. Then, all gateway services of the first gateway and the second gateway service are upgraded to obtain upgraded all gateway services and the upgraded second gateway service. For each upgraded gateway service in the upgraded all gateway services and the upgraded second gateway service, it can be determined from the database whether each upgraded gateway service has a client connection record. Finally, any upgraded gateway service without a client connection record is deleted. For example, if the first gateway originally had 10 gateway services, a new gateway service is created, and then all 11 gateway services are upgraded. After the upgrade is completed, it is determined from the database whether each upgraded gateway service has a client connection record. If only one gateway service has no client connection record, this gateway service can be deleted; if multiple gateway services have no client connection record, any one of them can be deleted.

[0151] By implementing the above process, during the gateway upgrade, a second gateway service is created in advance, all gateway services are upgraded, and finally, client connection records are checked and an unused gateway service is cleaned up. This ensures that the gateway runs efficiently and stably after the upgrade, improving resource utilization efficiency.

[0152] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0153] A basic host security system is a client-server architecture system. It installs an agent into the user's host environment to perform basic data detection and reporting. The agent typically collects data such as processes, traffic, and login activity on the machine. The backend uses rule matching and sample analysis to identify malicious operations and files in the user's environment, and then notifies the user to take action via alerts and support tickets. See also Figure 7 , Figure 7This is a schematic diagram of the architecture of a host security system provided in this application embodiment. It includes a user interface (UI) 701, a server 702, a database 703 (DB), and multiple clients. Actors represent users of the host security system or any external entity interacting with the system. The UI is responsible for interacting with the user, the server is responsible for business logic processing, the database is responsible for data storage, and the client is responsible for executing specific tasks. Specifically: First, the server can start a backend response service to detect a fixed port so that agents can connect to this port. Then, the agent is installed in the user's environment. After each agent starts, it actively connects to the backend service by obtaining the server address configured locally, establishing a long TCP connection. The main interactions between the server and clients include: asset reporting in the agent's operating environment, processes, traffic, logins, etc.; issuing server commands; and reporting agent detection results and real-time data.

[0154] Figure 7 The host security system in this context has a simple structure and low maintenance cost, and can function normally with thousands of machine connections. However, as data volume increases and business logic becomes more complex, it suffers from the following drawbacks: the server has a single point of failure and cannot scale horizontally; it cannot handle sudden surges in traffic, and when the data volume becomes large, the host security system may become sluggish or even crash, while when business operations are idle, the host security system consumes very little resources, resulting in wasted service resources; typical structured data warehouses cannot support data storage of hundreds of millions of records; and the logic of the host security system is implemented in a single module, resulting in high coupling.

[0155] The data processing method proposed in this application can solve the above problems.

[0156] I. Eliminate single points of failure and enable horizontal scaling of services.

[0157] See Figure 8 , Figure 8This is a schematic diagram of a horizontally scaled server architecture provided in this application embodiment. The naming service 802 provides dynamic routing functionality, capable of real-time detection of server online / offline status and anomalies, providing service discovery capabilities for each client. It should be noted that one Server in the backend service layer 803 can connect to multiple gateways in the access layer 804, and one gateway in the access layer 804 can connect to multiple Agents. The backend service layer 803 includes: Server 1, Server 2, and Server N. The backend service layer (i.e., the servers) is responsible for handling business logic, data processing, and interaction with the database 801. The access layer 804 includes multiple gateways (i.e., a gateway cluster), which are responsible for request routing, load balancing, security control, and other functions. Gateways can forward requests to appropriate backend servers based on the request type or target. See also... Figure 9 , Figure 9 This is a schematic diagram of the client access process provided in this application embodiment. Step S901: Access layer registration information. After the gateway in the access layer starts, it registers its local information with the naming service. Step S902: Access layer sends a heartbeat to the naming service. The gateway in the access layer can maintain the heartbeat and start the service detection port. Step S903: Client requests server Internet Protocol address. When the Agent (i.e., the target client) starts, the gateway reads its local configuration, requests the server Internet Protocol address, obtains the service domain name, and uploads the service domain name to the naming service. The naming service then returns an available gateway IP and port (i.e., the first network parameter) to the Agent. Step S904: Client establishes a connection with the access layer. To ensure balanced connections to the backend, the naming service needs to be configured with a weighted random allocation algorithm (e.g., the first Agent connects to the first gateway in the access layer, the next Agent connects to another gateway, and so on). After obtaining the IP and port of a gateway in the access layer, the Agent initiates a connection request to the gateway. S905, The access layer successfully establishes a connection with the client. After the connection is established, proceed to step S906, where the client sends a token to the access layer for authentication. Step S907, The access layer verifies the token. Once the token is validated, the client can maintain a persistent TCP connection with the gateway. Step S908, The client sends a heartbeat to the access layer. Step S909, The client reports asset information to the access layer. Step S910, The access layer issues a task to the client.

[0158] After an Agent is authenticated, the gateway writes the Agent's ID and the gateway's IP address (i.e., the mapping relationship) to a database (e.g., a remote dictionary service (Redis)). When the backend service layer needs to issue a command to a specified Agent, it can first query Redis to find the gateway connected to that Agent, send a command message to that gateway, and then the gateway sends a command message to that Agent, thus sending the command message to the specified Agent.

[0159] When a user restarts their machine, it triggers the Agent to reconnect to the gateway. If, during the reconnection, the gateway IP (i.e., the second network parameter) requested by the Agent is different from the one before the reconnection, it will result in two connection records for the Agent in Redis, causing a failure. To avoid this failure, when the gateway detects that the Agent has disconnected, it can delete the connection information in Redis. After the Agent successfully reconnects, before setting the Redis record, it should first iterate through all the connection information in Redis and delete the connection record for that Agent before the reconnection.

[0160] In real-world scenarios, during agent upgrades, there are instances where the new agent is started first, followed by the deletion of the old agent. Because the disconnection operation is performed last, and the agent's connection record in Redis is deleted last, commands sent from the gateway to the specified agent may fail. Therefore, when recording agent connection information, the agent's connection port should also be recorded. This ensures that multiple connections to the same agent do not use the same port.

[0161] During the gateway service upgrade process, the gateway service process (i.e., the gateway service itself) will exit and restart. The naming service can set this background process to an abnormal state. If a previously connected Agent socket is disconnected, when the gateway service reconnects to the Agent, the gateway is not running and cannot obtain its IP address. Agents will then connect to existing gateways in a balanced manner. This process repeats until the upgrade is complete. The first upgraded gateway service will participate in the most Agent reconnections and connect to the most Agents, while the last upgraded gateway service will be idle with no Agent connections. There are three possible handling methods: 1) When an Agent detects a disconnected socket, it should not immediately trigger a reconnection. Instead, it should wait for a pre-set time until the gateway service starts up before triggering a reconnection. The gateway service should have a short startup time, ideally within seconds, so that it can immediately participate in the reconnection of existing Agents. 2) The gateway service can use Kubernetes' upgrade capabilities to start a new process (the upgraded first gateway service) and then shut down the old process (the un-upgraded first gateway service). Configure the gateway service update strategy to RollingUpdate, maxUnavailable to 0, and maxSurge to 1. During deployment, the gateway will first create a new Pod, then delete an old Pod, then create another new Pod, and then delete another old Pod, ensuring the gateway service remains available at all times. 3) Reserve an extra process (i.e., a second gateway service). For example, if the gateway service has 200 processes, but 201 are actually deployed during the upgrade, after the upgrade, checking the Redis connection records will reveal that process number 201 has no agent connection. Processes without agent connections can be deleted.

[0162] After multiple rounds of backend and agent upgrades, although the naming service uses a random weighted allocation algorithm, it doesn't record the specific number of service connections, failing to achieve true even distribution. This leads to an imbalance in connections, with the gateway that was upgraded first having the most connections. There are three solutions: 1) Limit the maximum number of connections per Pod; exceeding this limit returns an error, and the agent will attempt to connect to other backend services. 2) Use a hash ring service allocation algorithm. This ensures that even if the agent restarts, it will connect to the previous server; however, this requires good randomness in the agent's ID. 3) Replace Redis and the naming service with a single router service. The router service records all connection information. Before the agent starts, it retrieves connection information from the router service, determines the number of connections for all gateways, and returns the gateway service with the fewest connections to the agent.

[0163] The three solutions described above can be analyzed in the context of scaling scenarios. Since the existing servers have a large number of connections and are under heavy traffic, they cannot support all requests from the Agents. Scaling up is generally achieved by increasing the number of servers. Solution 2, after scaling, requires manually releasing connections on busy servers and triggering reconnections because existing Agent connections are not disconnected; only newly added Agents can connect to the expanded servers. Therefore, Solution 2 cannot achieve automatic scaling. Solution 3 offers the best performance but comes at a higher cost. The Router service replaces the naming service and, in addition to allocating server IPs based on the number of connections, needs to support dynamic server discovery and other functions. Solution 1 is easy to implement, has the lowest cost, and offers good performance. The maximum number of connections per Pod can be determined based on experience and the scenario, and should be dynamically configurable.

[0164] exist Figure 8 In this system, an access layer is introduced. This layer is specifically designed to manage the connection between the server and the agent, decoupling the connection client and the access client. In scenarios with a small-scale host security system, the access client and the backend service logic can be placed together, simplifying the host security system's logic. In scenarios with large business volumes and multiple servers processing the same data, the access layer can store the data in a message middleware, decoupling the connection between the agent and the backend service, and solving the data buffering problem during peak periods.

[0165] 2. Waveform peak reduction and smoothing.

[0166] Business traffic is typically unstable; during peak periods, traffic can be several times higher than usual. Placing excess data in a message queue and consuming it during off-peak hours can smooth out traffic spikes and prevent backend services from being overwhelmed by excessive traffic. There are many ways to implement message queues, such as Kafka, RabbitMQ, RocketMQ, and Pulsar. The following explanation uses Kafka as an example. See also... Figure 10 , Figure 10 This is a schematic diagram illustrating the data storage implementation provided in this application embodiment. First, the gateway 1001 reads data from the connection socket of the client 1002. Except for data requiring a response from the gateway itself (heartbeat, access, etc.), other data is not processed immediately but is first categorized and written to the message queue Kafka 1003 (i.e., a set of message queues). Different types of messages can be placed in different topics (i.e., message queues corresponding to the target data category). Then, the backend service layer consumes data from different topics as needed. To improve processing efficiency, the data consumed by the Kafka receiver 1004 is not processed directly but stored in a queue 1005 (Channel) within the program. The data processing thread 1006 determines the processing logic for each message, and then a dedicated processing routine retrieves data from the Channel for processing. A routine, similar to a thread, is the smallest unit of concurrent processing in a program. When the amount of data in the Channel accumulates to a certain level, it means the processing flow is relatively busy, at which point the Kafka consumption program needs to be paused.

[0167] Message queues decouple the backend service layer from message processing routines. Routines can retrieve data from the message queue at any time for processing without caring about the source of the data. They can also shuffle the order of messages and the backend service layer. This makes them suitable for stateless scenarios where message order is not critical.

[0168] 3. Use distributed data warehouses to process big data.

[0169] Traditional data storage uses relational databases, based on the relational model. Data is stored in tabular form and manipulated using Structured Query Language (SQL). Data is stored in tables on a single server, emphasizing data consistency and transaction support, making it suitable for small-scale applications and host-based security systems with high data consistency requirements. However, in big data scenarios, when tables contain hundreds of millions of records, storage and retrieval become slow, and relational databases cannot meet the demands. Even with database clusters (such as Oracle clusters), the number of records that can be handled is limited, and they become unsustainable once a certain scale is reached.

[0170] This application uses a distributed data warehouse for processing, distributing data across multiple computers and managing and querying it via network connections. Distributed databases offer higher availability and fault tolerance; even if one node fails, the database system can still function normally. Distributed databases can improve query performance through parallel processing and load balancing, while the query performance of relational databases is limited by the processing power of a single server. Distributed data warehouses increase data storage capacity but also increase the cost of using the data.

[0171] See Figure 11 , Figure 11 This is another schematic diagram illustrating the data storage implementation provided in this application embodiment. First, the business server 1101 integrates a data conversion software development kit 1102 (SDK) for formatting data. The data conversion SDK can be used to establish communication services with the data conversion client 1103, communicating through a pre-set protocol (e.g., TCP protocol). Second, the data conversion client 1103 sends the collected data to a message queue in the data bank 1104 for temporary storage. The task scheduling system 1105 reads data from the message queue in the data bank 1104. Then, the task scheduling system 1105 can perform rule-based parsing (equivalent to a filtering process, i.e., a preset parsing rule) on the data in the message queue and persist the parsed data to the data warehouse 1106 (DW) (i.e., distributed data storage). Finally, the index 1107 (Idex) can be a component for reading data. The Idex supports offline data reading and analysis from the DW in a manner similar to SQL.

[0172] Fourth, separate the primary and secondary tasks, and divide and conquer.

[0173] Putting all business data into a message queue may not be the best option, especially for data that needs to be processed quickly. While message queues offer advantages such as decoupling and asynchronous processing, they can introduce unacceptable latency in certain scenarios. Therefore, classifying data and distinguishing between the general data plane and the control plane is a wise approach.

[0174] Normal data surface: Normal data surface typically refers to data that does not have high real-time requirements and can tolerate latency. Examples include logs, files, and data reported by commands executed by the agent. Normal data surface can be processed asynchronously by placing it in a message queue. Then, batch processing or scheduled tasks can be used to centrally process the normal data surface to improve efficiency.

[0175] Control plane definition: Control plane data refers to data with extremely high real-time requirements that needs to be processed immediately (this can include data generated by commands executed by the Agent under the instruction of the background service layer). Examples include task status reporting, emergency alarms, and real-time detection data. Control plane data can be processed through synchronous calls or fast channels to ensure timely response and processing. Dedicated real-time processing frameworks or systems can be used to process control plane data.

[0176] See Figure 12 , Figure 12 This is another architectural diagram of the host security system provided in the embodiments of this application, such as... Figure 12 As shown, the control plane responds in real time to agent access and exit, maintaining asset information for the entire host security system. Based on this asset information, system functions such as publishing, task assignment, data archiving, and data distribution can be implemented. Since the ordinary data plane contains significantly more data than the control plane, subsequent storage processing can be performed using a data transformation SDK when data storage is required. For specific implementation details, please refer to [link / reference]. Figure 10 The details of this will not be elaborated here. Since the control plane data volume is relatively small, it can be stored in a regular data warehouse.

[0177] The control plane processing logic includes the release module 1203, the heartbeat module 1204, and the task module 1205.

[0178] The release module 1203 can interface with the release platform. The backend service layer can transmit release (i.e., upgrade) related information to the client 1202, which then executes the specific release action. After execution, the client 1202 sends the execution result to the gateway 1201, which then reports the result back to the backend service layer. For example, to upgrade a component, the backend service layer can package the component's server address and other related information and send it to the Agent. Upon receiving this information, the Agent can download the target version of the component, upgrade to the target version, and return the release result to the gateway, which then sends the result back to the release module. The heartbeat module 1204 is used to sense the status of the client 1202. The task module 1205 can be used to issue tasks. For example, if a component (e.g., Python 2.7) in an Agent has a problem, and it's necessary to determine which Agents have the same Python 2.7 version issue, a task can be issued to the Agent to query the Python version, and the Agent will then report the query result.

[0179] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 455 in the memory 450 may include: a command sending module 4551, used to send a task command to the target client through the first gateway in response to the establishment of a connection between the first gateway in the gateway cluster and the target client among multiple clients; a data receiving module 4552, used to receive target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; a determination module 4553, used to determine the data type of the target data; and a data processing module 4554, used to process the target data according to the data processing method corresponding to the data type.

[0180] In some embodiments, the command sending module 4551 is further configured to, in response to generating the command message of the task command, obtain the client identifier of the target client; determine the first gateway that has established a connection with the target client from the database based on the mapping relationship between the gateway identifier of the first gateway and the client identifier of the target client; send the command message of the task command to the first gateway; and send the command message of the task command to the target client through the first gateway.

[0181] In some embodiments, the command sending module 4551 is further configured to, in response to a target client among the plurality of clients starting up, determine the first gateway from the gateway cluster; send the first network parameters of the first gateway to the target client; receive a first connection request sent by the target client according to the first network parameters; the first connection request includes a gateway identifier of the first gateway; in response to the first connection request, control the target client to connect to the first gateway; and map the gateway identifier to the client identifier and store it in the database.

[0182] In some embodiments, the command sending module 4551 is further configured to: delete the mapping relationship between the gateway identifier and the client identifier from the database in response to the target client disconnecting from the first gateway; determine a second gateway from the gateway cluster in response to the target client re-establishing a connection after disconnecting from the first gateway; send the second network parameters of the second gateway to the target client; receive a second connection request sent by the target client according to the second network parameters; the second connection request includes the gateway identifier of the second gateway; control the target client to connect to the second gateway in response to the second connection request; and map the gateway identifier of the second gateway to the client identifier and store it in the database.

[0183] In some embodiments, the data processing module 4554 is further configured to: determine the target data category of the target data in response to the target data being of a first real-time type; obtain the target data from a message queue corresponding to the target data category; wherein the real-time indicator value of the target data having the first real-time type is less than a preset real-time indicator value threshold; determine the target data processing thread corresponding to the target data category; and call the target data processing thread to perform data processing on the target data.

[0184] In some embodiments, the data processing module 4554 is further configured to: in response to the existence of a message queue corresponding to the target data category in the current message queue set, store the target data in the message queue corresponding to the target data category; in response to the absence of a message queue corresponding to the target data category in the current message queue set, create a message queue corresponding to the target data category in the message queue set, and store the target data in the message queue corresponding to the target data category.

[0185] In some embodiments, the data processing module 4554 is further configured to parse the target data in the message queue according to a preset parsing rule to obtain parsed data; determine a target data storage unit in a preset distributed data storage; and store the parsed data in the target data storage unit.

[0186] In some embodiments, the apparatus further includes a gateway upgrade module, configured to, in response to the gateway service of the first gateway being in an upgrade state, invoke a timer to perform timing; and, in response to the timer timing equaling a preset duration, re-establish the connection between the target client and the first gateway.

[0187] In some embodiments, the gateway upgrade module is further configured to upgrade the first gateway service of the first gateway in response to obtain the upgraded first gateway service, and realize the connection between the first gateway and the target client through the upgraded first gateway service; the first gateway service is the gateway service currently used to realize the connection between the first gateway and the target client; and the first gateway service before the upgrade is terminated.

[0188] In some embodiments, the gateway upgrade module is further configured to: create a second gateway service for the first gateway; upgrade all gateway services of the first gateway and the second gateway service to obtain upgraded all gateway services and upgraded second gateway services; for each upgraded gateway service among the upgraded all gateway services and the upgraded second gateway service, determine whether each upgraded gateway service has a client connection record; and delete any upgraded gateway service from the upgraded gateway services that do not have a client connection record.

[0189] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0190] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application. For example, ... Figure 3 The data processing method is shown.

[0191] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0192] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0193] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0194] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0195] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: In response to the establishment of a connection between the first gateway in the gateway cluster and the target client among multiple clients, a task command is sent to the target client through the first gateway based on the mapping relationship between the first gateway and the target client; Receive target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; Determine the data type of the target data; The target data is processed according to the data processing method corresponding to the data type.

2. The method according to claim 1, characterized in that, The step of sending task commands to the target client through the first gateway based on the mapping relationship between the first gateway and the target client includes: In response to the command message that generates the task command, obtain the client identifier of the target client; Based on the mapping relationship between the gateway identifier of the first gateway and the client identifier of the target client, the first gateway that establishes a connection with the target client is determined from the database; Send the task command message to the first gateway; The first gateway sends a command message containing the task command to the target client.

3. The method according to claim 2, characterized in that, Before obtaining the client identifier of the target client in response to the command message that generates the task command, the method further includes: In response to the launch of a target client among the plurality of clients, the first gateway is determined from the gateway cluster; Send the first network parameters of the first gateway to the target client; Receive a first connection request sent by the target client based on the first network parameters; the first connection request includes the gateway identifier of the first gateway; In response to the first connection request, control the target client to connect to the first gateway; The gateway identifier is mapped to the client identifier and then stored in the database.

4. The method according to claim 3, characterized in that, The method further includes: In response to the target client disconnecting from the first gateway, the mapping relationship between the gateway identifier and the client identifier is deleted from the database; In response to the re-establishment of a connection between the target client and the first gateway after the connection is lost, a second gateway is determined from the gateway cluster; Send the second network parameters of the second gateway to the target client; Receive a second connection request sent by the target client based on the second network parameters; the second connection request includes the gateway identifier of the second gateway; In response to the second connection request, control the target client to connect to the second gateway; The gateway identifier of the second gateway is mapped to the client identifier and then stored in the database.

5. The method according to claim 1, characterized in that, The step of processing the target data according to the data processing method corresponding to the data type includes: In response to the fact that the data type of the target data is a first real-time type, the target data category of the target data is determined; The target data is obtained from the message queue corresponding to the target data category; wherein the real-time indicator value of the target data having the first real-time type is less than a preset real-time indicator value threshold. Determine the target data processing thread corresponding to the target data category; The target data processing thread is invoked to process the target data.

6. The method according to claim 5, characterized in that, Before retrieving the target data from the message queue corresponding to the target data category, the method further includes: In response to the existence of a message queue corresponding to the target data category in the current message queue set, the target data is stored in the message queue corresponding to the target data category; If no message queue corresponding to the target data category exists in the current message queue set, a message queue corresponding to the target data category is created in the message queue set, and the target data is stored in the message queue corresponding to the target data category.

7. The method according to claim 5, characterized in that, After storing the target data in the message queue corresponding to the target data category, the method further includes: According to the preset parsing rules, the target data in the message queue is parsed to obtain parsed data; Determine the target data storage unit in the pre-defined distributed data storage; The parsed data is stored in the target data storage unit.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to the gateway service of the first gateway being in an upgrade state, a timer is invoked to start timing; In response to the timer reaching a preset duration, the connection between the target client and the first gateway is re-established.

9. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to upgrading the first gateway service of the first gateway, an upgraded first gateway service is obtained, and the connection between the first gateway and the target client is realized through the upgraded first gateway service; the first gateway service is the gateway service currently used to realize the connection between the first gateway and the target client. The first gateway service prior to the upgrade was terminated.

10. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Create a second gateway service for the first gateway; Upgrade all gateway services of the first gateway and the second gateway service to obtain upgraded all gateway services and upgraded second gateway services; For each upgraded gateway service, including all upgraded gateway services and each upgraded second gateway service, determine whether each upgraded gateway service has a client connection record. Remove any upgraded gateway service from the list of upgraded gateway services that do not have client connection records.

11. A data processing apparatus, characterized in that, The device includes: The command sending module is used to respond to the establishment of a connection between the first gateway in the gateway cluster and the target client among multiple clients, and to send task commands to the target client through the first gateway based on the mapping relationship between the first gateway and the target client. A data receiving module is used to receive target data sent by the first gateway; wherein the target data is data generated by the target client in response to the task command, and the target data is stored by the first gateway; The determining module is used to determine the data type of the target data; The data processing module is used to process the target data according to the data processing method corresponding to the data type.

12. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 10.

Images