Data processing method, system, device, computer device and storage medium

Abstract

The application relates to a data processing method, system and device, computer equipment, a computer readable storage medium and a computer program product. The data processing method is applied to a data center equipment; the data center equipment is deployed with a business processing process, a data reporting interface and a data proxy process. The method comprises the following steps: in response to a calling event of the business processing process to the data reporting interface, obtaining data information reported by the business processing process through the data reporting interface; writing the data information into a shared storage space; calling the data proxy process according to a time period matched by the business processing process, reading at least one piece of data information written in the time period from the shared storage space; and reporting the at least one piece of data information to a data server. The data information satisfies a data reporting condition configured for the data reporting interface; and the shared storage space is a storage space shared by the business processing process and the data proxy process. The above method can improve work efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, system, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, Internet Data Centers (IDCs) have emerged, providing services such as server hosting, network connectivity, and data storage. In an IDC environment, it is necessary to collect data from each data center device (IDC device) deployed within it, including but not limited to hardware performance data such as memory, CPU (Central Processing Unit), and hard disk size, as well as business interaction data such as remote logs and management data.

[0003] In traditional technology, data center devices report data centrally by calling the data reporting interface of a reporting server. However, this traditional approach involves frequent calls to the reporting server's interface from multiple data center devices, which can easily lead to lag due to heavy load and low efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, system, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve work efficiency in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method applied to a data center device; the data center device is equipped with a business processing process, a data reporting interface, and a data proxy process; the method includes:

[0006] In response to the business processing process's call event to the data reporting interface, the system obtains the data information reported by the business processing process through the data reporting interface; the data information satisfies the data reporting conditions configured for the data reporting interface.

[0007] The data information is written into a shared storage space; the shared storage space is a storage space shared by the business processing process and the data proxy process.

[0008] According to the time period matched by the business processing process, the data proxy process is invoked to read at least one piece of data information written within the time period from the shared storage space;

[0009] The at least one piece of data information shall be reported to the data server.

[0010] Secondly, this application also provides a data processing system. The system includes data center equipment and a data server; the data center equipment is deployed with business processing processes, data reporting interfaces, and data proxy processes.

[0011] The data center equipment is configured to respond to a call event from the business processing process to the data reporting interface, obtain data information reported by the business processing process through the data reporting interface; write the data information into a shared storage space; call the data proxy process according to a time period matched by the business processing process, and read at least one piece of data information written within the time period from the shared storage space; report the at least one piece of data information to the data server; the data information satisfies the data reporting conditions configured for the data reporting interface; the shared storage space is a storage space shared by the business processing process and the data proxy process.

[0012] The data server is used to store the data information reported by the data center equipment.

[0013] Thirdly, this application also provides a data processing apparatus for use in data center equipment; the data center equipment is equipped with a business processing process, a data reporting interface, and a data proxy process; the apparatus includes:

[0014] The data acquisition module is used to respond to the call event of the business processing process to the data reporting interface, and acquire the data information reported by the business processing process through the data reporting interface; the data information satisfies the data reporting conditions configured for the data reporting interface.

[0015] A shared storage module is used to write the data information into a shared storage space; the shared storage space is a storage space shared by the business processing process and the data proxy process.

[0016] The data reading module is used to call the data agent process according to the time period matched by the business processing process, and read at least one piece of data information written within the time period from the shared storage space;

[0017] The data reporting module is used to report the at least one piece of data information to the data server.

[0018] In one embodiment, the data reporting conditions are characterized by data range and data format; the device further includes a call event triggering module, used for:

[0019] When the raw data generated during the business process is obtained, selected data that meets the data range is filtered from the raw data;

[0020] When the selected data is converted to a format that conforms to the data format, a call event is triggered in the business processing process for the data reporting interface.

[0021] In one embodiment, the device further includes:

[0022] The lock addition module is used to add the mutex lock to the data information when there is an idle mutex lock, and change the state of the mutex lock to the locked state;

[0023] The lock release module is used to release the mutex lock and change the state of the mutex lock to an idle state when the data information is written to the shared storage space.

[0024] In one embodiment, the data reading module is specifically used for:

[0025] A timer configured for the business processing process is determined; the timer is triggered according to a time period matching the business processing process.

[0026] When the timer is triggered, the data agent process is invoked to read at least one piece of data information written within the time period from the shared storage space.

[0027] In one embodiment, the data reporting module is specifically used for:

[0028] The read data information is validated to obtain the validation result;

[0029] If the verification result is successful, the data information is reported to the data server through the proxy server.

[0030] In one embodiment, the number of data servers is multiple; the proxy server specifically routes data according to the information identifier carried by each data piece of information, and reports each data piece of information to the corresponding data server. In this embodiment, the device further includes an information identifier determination module, used for:

[0031] Determine the data type of the data information;

[0032] When the data type is business interaction data, the information identifier carried by the data information is determined based on the business party that interacts with the data center equipment.

[0033] When the data type is hardware performance data, the device identifier of the data center device is determined as the information identifier carried by the data information.

[0034] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0035] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0036] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0037] The aforementioned data processing methods, systems, devices, computer equipment, computer-readable storage media, and computer program products deploy and configure separate data reporting interfaces with data reporting conditions within the data center equipment. Data reporting is achieved through a data proxy process interconnected with the data server. This means each data reporting interface only needs to handle its own local data reporting, reducing the load on individual interfaces and improving work efficiency. Furthermore, the data center equipment is configured with shared storage space shared by the business processing process and the data proxy process, allowing for the simultaneous writing and reading of data information, further improving efficiency. By calling the data proxy process to read data from the shared storage space according to the time period matched by the business processing process, batch processing of data information can be achieved while ensuring real-time data reporting, further improving work efficiency. Therefore, using the data processing method of this application can improve work efficiency during the process of data center equipment reporting data information to the data server. Attached Figure Description

[0038] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0039] Figure 2 This is a schematic diagram illustrating the data reporting principle of a data center device in one embodiment;

[0040] Figure 3 This is a flowchart illustrating a data processing method in one embodiment;

[0041] Figure 4 This is a flowchart illustrating the data processing method in another embodiment;

[0042] Figure 5 This is a schematic diagram of the composition of a data processing system in one embodiment;

[0043] Figure 6This is a schematic diagram of the composition of the data processing system in another embodiment;

[0044] Figure 7 This is a schematic diagram illustrating the connection relationship between the data proxy process and the proxy server in one embodiment;

[0045] Figure 8 This is a schematic diagram of the composition of a data center device in one embodiment;

[0046] Figure 9 This is a schematic diagram of the data routing process in one embodiment;

[0047] Figure 10 This is a structural block diagram of a data processing device in one embodiment;

[0048] Figure 11 This is an internal structural diagram of a computer device in one embodiment;

[0049] Figure 12 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, multiple data center devices 102 of an internet data center communicate with multiple data servers 104 via a network. This communication network can be a wired network or a wireless network. Therefore, the data center devices 102 and the data servers 104 can be directly or indirectly connected via wired or wireless communication. For example, the data center devices 102 can be indirectly connected to the data servers 104 via a wireless access point, or the data center devices 102 can be directly connected to the data servers 104 via the Internet; this application does not impose any limitations on this.

[0052] Specifically, data center equipment 102 can be storage devices, network devices, servers, power supplies, or cooling equipment, etc. Data server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system can store the data that data server 104 needs to process. The data storage system can be set up independently, integrated into data server 104, or placed in the cloud or on other servers.

[0053] The data processing method in this embodiment can be executed independently by the data center device 102. For example... Figure 2 As shown, the data center device 102 deploys a business processing process 201, a data reporting interface 202, and a data proxy process 203. During data processing: in response to a call event from the business processing process 201 to the data reporting interface 202, it obtains the data information reported by the business processing process 201 through the data reporting interface 202; writes the data information to the shared storage space; according to the time period matched by the business processing process 201, it calls the data proxy process 203 to read at least one piece of data information written within the time period from the shared storage space; and reports at least one piece of data information to the data server 104. The data information satisfies the data reporting conditions configured for the data reporting interface 202; the shared storage space is shared by the business processing process 201 and the data proxy process 203. This storage space can be integrated on the data center device 102 or placed on other devices or servers.

[0054] In one embodiment, such as Figure 3 As shown, a data processing method is provided, which can be executed by a computer device, the computer device being... Figure 1 The data center device shown in the figure, the method includes the following steps:

[0055] Step S302: In response to the call event of the business processing process to the data reporting interface, obtain the data information reported by the business processing process through the data reporting interface.

[0056] In this context, a business processing process refers to a program or application that executes specific business logic within a computer system. This process is responsible for handling user requests, performing computational tasks, and monitoring the system. As a core component deployed on IDC (Internet Data Center) equipment, it is responsible for executing specific business logic. Because business processing processes handle a large amount of computation and data operations, they consume a significant portion of the IDC equipment's resources and generate data that needs to be reported. To gain a comprehensive understanding of the IDC equipment's operational status, the business processing process can collect data from the IDC equipment.

[0057] A data reporting interface is used for data reporting. This interface can be either a hardware interface or a software interface. In one optional embodiment, the data reporting interface is an API (Application Programming Interface). An API is a set of protocols and tools that define how different software components interact, allowing different software systems to communicate and exchange data through standardized interfaces. The data information meets the data reporting conditions configured for the data reporting interface. These conditions can be characterized by at least one of the following: data range, data format, and reporting time. Specifically, the data range specifies the content of the reported data and may include hardware performance data such as memory, CPU, and hard disk size, as well as business interaction data such as remote logs and management data. The data format specifies the format of the reported data and may include JSON (JavaScript Object Notation), XML (Extensible Markup Language), Protobuf (Protocol Buffers), and RDF (Resource Description Framework). The reporting time specifies the data reporting time.

[0058] In an IDC environment, data collection from all deployed data center equipment (102) is crucial for the operators of internet data centers. Timely collection and analysis of IDC equipment data not only allows for better management of hardware resources, optimized resource utilization, and improved overall operational efficiency, but also provides early warnings of problems and helps prevent large-scale failures, thereby enhancing the operational efficiency and service quality of internet data centers.

[0059] Specifically, during business processing, the business processing process can report data by calling the data reporting interface. The IDC device can respond to the business processing process's call to the data reporting interface and obtain the data information reported by the business processing process through the data reporting interface.

[0060] In an alternative embodiment, containerization technology can be used to deploy business processing processes for easier management and scaling. For example, container orchestration tools can be used to automate deployment, scaling, and fault recovery.

[0061] In an optional embodiment, the IDC device can deploy multiple service processing processes. Each process can report data by calling a data reporting interface during its service processing. The IDC device can respond sequentially according to the triggering order of the call events. When multiple processes simultaneously call the data reporting interface, the IDC device can respond sequentially based on the reporting priority of each process. The reporting priority can be determined based on the scope or importance of the business logic carried by the process. For example, regarding the first process responsible for system monitoring and the second process responsible for responding to user operations, the first process has a higher reporting priority because its monitoring results form the basis for decision-making and operation in the IDC environment, with a wider impact, while the second process's impact is limited to the currently responding user. Similarly, since security monitoring results are more important than traffic monitoring results, the reporting priority of the security monitoring process is higher than that of the traffic monitoring process.

[0062] In an optional embodiment, the business processing process can be configured with a robust error handling mechanism. In the event of a data reporting failure, the business processing process can record the interruption point in the data reporting process through logs and retry by making a second call to the data reporting interface. Thus, the data center equipment can respond to the business processing process's second call to the data reporting interface, determine the interruption point in the previous reporting process based on the log information, and then continue acquiring the data reported by the business processing process through the data reporting interface, starting from that interruption point. By recording the interruption point, retrying allows data transmission to continue from that point without having to re-upload the entire data message, further improving work efficiency.

[0063] Step S304: Write the data information into the shared storage space.

[0064] Shared storage is an inter-process communication (IPC) mechanism that allows multiple processes to share a storage area. Shared storage provides an efficient data transfer method without requiring data copying between processes. In this application, the shared storage space refers to the storage space shared by the business processing process and the data agent process. This shared storage space can be integrated into an IDC device or deployed in other devices. Optionally, a dedicated shared memory space can be allocated within the IDC device as the shared storage space for the business processing process and the data agent process. The data agent process is the process responsible for reading data from the shared storage space. Specifically, the data agent process can be a DataAgent.

[0065] Specifically, the data reported through the data reporting interface will be written to a shared storage space between the business processing process and the data proxy process. The business processing process is responsible for writing the data to the shared storage space, while the data proxy process is responsible for reading the data from the shared storage space. In an optional embodiment, the shared storage space uses a ring buffer or a linked list to improve read / write efficiency.

[0066] In some optional embodiments, synchronization mechanisms can be used to ensure data consistency and integrity in order to avoid data read / write conflicts. These synchronization mechanisms refer to data processing mechanisms that coordinate the execution order of threads or processes in a multi-threaded or multi-process environment. Specifically, these synchronization mechanisms may include semaphores, mutexes, and condition variables.

[0067] Step S306: According to the time period matched by the business processing process, call the data agent process to read at least one piece of data information written within the time period from the shared storage space.

[0068] As mentioned earlier, multiple business processing processes can be deployed in an IDC device, and each process carries different business logic and has different requirements for data reporting timeliness. Therefore, the IDC device can configure corresponding time periods for each business processing process. When a business processing process writes data to the shared storage space through the data reporting interface, the IDC device can call the data agent process to read data from the shared storage space according to the time period matched to that business processing process. The read data is at least one piece of data written to the shared storage space within that time period. Configuring the data reading period according to business needs ensures timely data reporting while avoiding the performance overhead of frequent reads.

[0069] In an optional embodiment, the IDC device can configure the same time period and shared storage space for business processing processes with similar data reporting timeliness requirements, based on the individual data reporting timeliness needs of its deployed multiple business processing processes. This ensures that multiple business processing processes can write data information to the shared storage space within a single time period, enabling the data agent process to read multiple data entries at a time, thus achieving batch reading and reporting of data information. The data reporting timeliness requirement can be characterized by data validity period, data update cycle, etc. Similar data reporting timeliness requirements between two business processing processes mean that the difference between the maximum data latency that each of the two business processing processes can receive is less than or equal to a difference threshold. Data latency refers to the time difference between the generation time of data information and the time it is reported to the data server.

[0070] In an optional embodiment, to prevent the same data information from being read repeatedly, the IDC device can delete the read data information in the shared storage space. The IDC device can also maintain the status of data information in the shared storage space. For example, when a data broker process reads data information, the IDC device can change the status of that data information from "pending read" to "read." Data information in the "read" state does not support read operations, but it does support write operations. Optionally, if the IDC device updates data information in the "read" state, it can change that data information from "read" to "pending read." The data update operation can be specifically executed by the business processing process.

[0071] Step S308: Report at least one piece of data to the data server.

[0072] A data server is a computer device specifically designed for storing, managing, and processing large amounts of data. During the data collection process targeting IDC (Internet Data Center) equipment, the collected data is ultimately reported to the data server.

[0073] The aforementioned data processing method deploys a separate data reporting interface configured with data reporting conditions within the data center equipment. Data reporting is then implemented through a data proxy process interconnected with the data server. This means each data reporting interface only needs to handle its own local data reporting, reducing the load on individual interfaces and improving efficiency. Furthermore, the data center equipment is configured with shared storage space shared by the business processing process and the data proxy process, allowing for the simultaneous writing and reading of data, further enhancing efficiency. By calling the data proxy process to read data from the shared storage space according to the time period matched by the business processing process, batch processing of data information can be achieved while ensuring real-time data reporting, further improving efficiency. Therefore, the data processing method of this application can improve efficiency during the process of data center equipment reporting data information to the data server.

[0074] In one embodiment, data reporting conditions are characterized by data range and data format. In this embodiment, the data processing method further includes: when the raw data generated during the business processing is obtained, selecting data that meets the data range from the raw data; and when the selected data is format-converted to obtain data information that conforms to the data format, triggering a call event for the data reporting interface by the business processing process.

[0075] The business processing process is used to perform business processing. In this embodiment, the business processing process refers to the process responsible for implementing the business processing procedure. The data scope is used to standardize the content of the reported data. The data format is used to standardize the format of the reported data. The raw data may include hardware performance data, business interaction data, environmental monitoring data, etc. Hardware performance data refers to various indicators and parameters used to measure the performance of computer hardware devices, specifically including the performance data of core components such as processors, memory, storage devices, and graphics cards. Business interaction data refers to various data generated in the business process, including log records and management data related to login, orders, users, products, and payments. Environmental monitoring data refers to data about the environment in which the IDC equipment is located, including temperature, humidity, etc.

[0076] Specifically, the business processing process can obtain the raw data generated during business processing, as well as the data range and data format configured for the data reporting interface. Then, the business processing process can filter selected data from the raw data that meets the data range to ensure that the content of the subsequently reported data matches the data reporting interface. Next, the business processing process can perform format conversion on the selected data to obtain data information that conforms to the specified data format. In this case, the IDC device will trigger a call event from the business processing process to the data reporting interface. Furthermore, the data range and data format can be predefined or defined in real-time when the business processing process obtains the raw data; this is not limited here. In an optional embodiment, the data reporting interface is an API interface with predefined data range and data format, which ensures the structured and standardized nature of the data.

[0077] In the above embodiments, the data reporting conditions are characterized by data format and data range. The data information to be reported can be limited in terms of both format and content, which can ensure the standardization of the reported data and facilitate subsequent data analysis.

[0078] In one embodiment, the data processing method further includes: when there is an idle mutex, adding a mutex to the data information and changing the state of the mutex to a locked state; when the data information is written to the shared storage space, releasing the mutex and changing the state of the mutex to an idle state.

[0079] A mutex is a synchronization mechanism used in multithreaded programming. Its main purpose is to ensure that only one thread can access a specific shared resource or code block at any given time. This mechanism is implemented through locking and unlocking operations to prevent multiple threads from accessing shared resources simultaneously, thereby avoiding data races and inconsistencies.

[0080] Specifically, before writing data to the shared storage space, the IDC device can determine whether an idle mutex exists. If an idle mutex exists, the IDC device can add a mutex to the data to be written and change the mutex's state to locked. Then, the data is written to the shared storage space while locked. After the data writing is complete, the mutex is released, and its state is changed back to idle. If no idle mutex exists, the device must wait for the mutex to be released before locking and data writing can proceed.

[0081] In an optional embodiment, invoking the data broker process to read at least one piece of data information written within a time period from the shared storage space includes: adding a mutex lock to the at least one piece of data information to be read in the shared storage space; invoking the data broker process to read the data information from the shared storage space while the mutex lock is engaged, and releasing the mutex lock.

[0082] In an optional embodiment, multiple mutexes can be deployed for the shared storage space. These mutexes are initialized and configured for inter-process sharing, allowing the business processing process and the data proxy process to use them during write or read operations, respectively. For example, when the business processing process writes data information A, the data proxy process can read data information from the shared storage space. Since data information A is locked, it cannot be read simultaneously. Therefore, the data information read by the data proxy process from the shared storage space may include data information B and C, but not data information A. Similarly, because mutexes are added, the read data information B and C cannot be simultaneously updated by the business processing process through write operations, thus ensuring data consistency.

[0083] In the above embodiments, by adding a mutex lock, it can be ensured that data information is not read when it is being written, thereby avoiding data read-write conflicts and improving data consistency and integrity.

[0084] In one embodiment, according to the time period matched by the business processing process, the data proxy process is invoked to read at least one piece of data information written within the time period from the shared storage space, including: determining the timer configured for the business processing process; and when the timer is triggered, invoking the data proxy process to read at least one piece of data information written within the time period from the shared storage space.

[0085] Specifically, timers configured for business processing processes are triggered according to a time period matching that of the business processing process. In practical applications, an IDC device can deploy multiple business processing processes, each configured with a corresponding timer. In other words, multiple timers can be deployed in an IDC device, and the relationship between business processing processes and timers is many-to-one.

[0086] Specifically, the IDC device can determine the timer corresponding to the business processing process from multiple configured timers. When this timer is triggered, it invokes the data proxy process to read data information from the shared storage space of the business processing process and the data proxy process. It can be understood that since the read action is triggered according to a time period, the data information read includes at least one piece of data written to the shared storage space within that time period. For example, taking a 1-minute time period as an example: data written to the shared storage space within the first minute will be read at the end of the first minute; data written to the shared storage space within the second minute will be read at the end of the second minute.

[0087] In the above embodiments, by configuring a timer for the business processing process and calling the data proxy process to read data information when the timer is triggered, it is possible to ensure that the timeliness meets the business processing requirements while avoiding the performance overhead caused by frequent reading, which is conducive to further improving work efficiency.

[0088] In one embodiment, reporting at least one piece of data to the data server includes: performing data verification on the read data to obtain a verification result; and if the verification result is successful, reporting the data to the data server through a proxy server.

[0089] A proxy server is an intermediary server that sits between IDC (Internet Data Center) equipment and data servers. The proxy server receives network requests from the IDC equipment and forwards them to the data servers, then returns the responses from the data servers to the IDC. Proxy servers play a crucial role in network security, performance optimization, privacy protection, and resource management. They not only improve network efficiency and security but also provide users with a more flexible and controllable network access experience.

[0090] Specifically, when an IDC device invokes a data broker process to read data from shared storage, it can perform data verification to determine the data's legitimacy. The specific verification methods are not unique; they may include parity checking, cyclic redundancy check, hash verification, etc. If the verification passes, the IDC device then reports the data to the data server through the broker server.

[0091] In an optional embodiment, data verification can be performed by a data proxy process. Specifically, after reading the data, the data proxy process can perform some simple preprocessing, including data verification and format conversion, to ensure the legality and consistency of the data. Furthermore, during data transmission, the data proxy process has a robust error handling and retry mechanism to ensure reliable data transmission.

[0092] In an optional embodiment, the connection between the IDC device and the proxy server is many-to-one. Since a proxy server needs to respond to data reporting requests from multiple IDC devices, data verification is performed before initiating a data reporting request to ensure the legality and integrity of the data. This can prevent abnormal data from affecting the normal operation of the proxy server, thereby ensuring the stable operation of the entire data processing system.

[0093] In the above embodiments, the read data information is verified, and if the verification is successful, the data information is reported to the data server through the proxy server. On the one hand, the legality and integrity of the data can be ensured through data verification, and on the other hand, the security and efficiency of the data transmission link can be improved by using the proxy server.

[0094] In one optional embodiment, there are multiple data servers. In this embodiment, the proxy server specifically routes data according to the information identifier carried by each data message, and reports each data message to the corresponding data server. The data processing method further includes: determining the data type of the data message; if the data type is business interaction data, determining the information identifier carried by the data message based on the business party interacting with the data center equipment; if the data type is hardware performance data, determining the device identifier of the data center equipment as the information identifier carried by the data message.

[0095] Data routing refers to the process of forwarding data from one node to another according to specific rules or algorithms. This routing algorithm can specifically include hash algorithms and consistent hashing algorithms. When there are multiple data servers, the proxy server can parse each piece of data to determine the information identifier carried by each piece of data. Then, the proxy server can route the data according to the information identifier, reporting each piece of data to the data server that matches the information identifier. For IDC devices, the information identifier carried by the data needs to be determined before the data is transmitted to the proxy server.

[0096] Specifically, the IDC device can determine the data type of the data information. This data type can include business interaction data and hardware performance data. Then, it determines the information identifier carried by the data information according to the data type. Optionally, when the data type is business interaction data, the IDC device can determine the information identifier carried by the data information based on the business party interacting with the data center equipment. For example, the IDC device can determine the business party identifier of the business party as the information identifier carried by the data information; it can also determine the geographical identifier of the business party's location as the information identifier carried by the data information. Optionally, when the data type is hardware performance data, the IDC device can determine the equipment identifier of the data center equipment as the information identifier carried by the data information.

[0097] In one possible implementation, the business processing process can determine the information identifier carried by the data information before reporting the data information through the data reporting interface to ensure the accuracy of the information identifier.

[0098] In the above embodiments, the information identifier carried by the data information is determined according to the data type, and the data is routed according to the information identifier to determine the reporting server of the data information. This ensures that data information from the same business party or the same IDC device will be routed to the same data server, which facilitates centralized data processing by the subsequent data server and helps to further improve work efficiency.

[0099] In one embodiment, such as Figure 4 As shown, a data processing method is provided, which can be executed by a computer device, the computer device being... Figure 1 The data center equipment shown is equipped with service processing processes, data reporting interfaces, and data proxy processes. In this embodiment, the method includes the following steps:

[0100] Step S401: Obtain the data reporting conditions configured for the data reporting interface;

[0101] Among them, data reporting conditions are characterized by data range and data format;

[0102] Step S402: If the raw data generated during the business processing is obtained, select data that meets the data range is filtered from the raw data.

[0103] The business processing procedure is executed by the business processing process.

[0104] Step S403: After converting the selected data into a format and obtaining data information that conforms to the data format, the business processing process is triggered to call the data reporting interface.

[0105] Step S404: Determine the data type of the data information;

[0106] Step S405: When the data type is business interaction data, determine the information identifier carried by the data information based on the business party that interacts with the data center equipment.

[0107] Step S406: When the data type is hardware performance data, the device identifier of the data center device is determined as the information identifier carried by the data information.

[0108] Step S407: In response to the call event of the business processing process to the data reporting interface, obtain the data information reported by the business processing process through the data reporting interface;

[0109] Step S408: If there is an idle mutex lock, add a mutex lock to the data information and change the state of the mutex lock to the locked state;

[0110] Step S409: While the device is locked, write the data information to the shared storage space.

[0111] Among them, the shared storage space is the storage space shared by the business processing process and the data proxy process;

[0112] Step S410: When data information is written to the shared storage space, release the mutex lock and change the state of the mutex lock to the idle state;

[0113] Step S411: Determine the timer configured for the business processing process;

[0114] The timer is triggered according to a time period that matches the business processing process.

[0115] Step S412: When the timer configured for the business processing process is triggered, the data agent process is invoked to read at least one piece of data information written within the time period from the shared storage space.

[0116] Step S413: Perform data verification on the read data information to obtain the verification result;

[0117] Step S414: If the verification result is successful, the data information is reported to the data server through the proxy server.

[0118] The proxy server is used to route data according to the information identifier carried by each data information, and report each data information to the data server that matches the information identifier.

[0119] The data processing method described above, through the collaborative work of the business processing process, data reporting interface and data proxy process deployed on the IDC equipment, can not only reduce the load on a single interface in the case of large-scale data reporting in the IDC environment, but also achieve batch processing of data information while ensuring the real-time nature of data reporting, which is conducive to improving work efficiency.

[0120] On the other hand, such as Figure 5 As shown, this application also provides a data processing system, including a data center device 501 and a data server 503. The data center device 501 deploys a business processing process, a data reporting interface, and a data proxy process. The data center device 501 is used to: respond to a call event from the business processing process to the data reporting interface, obtain data information reported by the business processing process through the data reporting interface; write the data information to a shared storage space; according to a time period matched by the business processing process, call the data proxy process to read at least one piece of data information written within the time period from the shared storage space; and report at least one piece of data information to the data server. The data server 503 is used to store the data information reported by the data center device 501.

[0121] The data reported by the business processing process through the data reporting interface meets the data reporting conditions configured for the data reporting interface. Shared storage space is shared by the business processing process and the data proxy process.

[0122] Specifically, the business processing process running on the IDC equipment executes specific business logic, generating data that needs to be reported during execution. Optionally, the business processing process can format the generated data according to a predefined format to ensure data structure and standardization, facilitating subsequent processing. Optionally, the business processing process can call the data reporting interface to write the formatted data information into a shared storage space. This shared storage space is shared by the business processing process and the data agent process. On the other hand, the data agent process can be triggered at set time intervals to read the data information written by the business processing process from the shared storage space. Optionally, the data agent process can perform data verification during data reading to ensure the legality and integrity of the data. The data information read by the data agent process will be reported to the data server, where it will be stored.

[0123] In an optional embodiment, data reporting conditions are characterized by data range and data format. In this embodiment, the data center device is further configured to: upon obtaining raw data generated during business processing, filter selected data from the raw data that meets the data range; and, upon converting the selected data to a format to obtain data information conforming to the data format, trigger a call event from the business processing process to the data reporting interface.

[0124] In an optional embodiment, the data center device is further configured to: add a mutex lock to data information and change the state of the mutex lock to a locked state when an idle mutex lock exists; and release the mutex lock and change the state of the mutex lock to an idle state when the data information is written to the shared storage space.

[0125] In an optional embodiment, the data center device is specifically configured to: determine a timer configured for a business processing process; and, when the timer is triggered, invoke a data broker process to read at least one piece of data information written within the time period from the shared storage space. The timer configured for the business processing process is triggered according to a time period matching the business processing process.

[0126] In an optional embodiment, the data center device is specifically used to: perform data verification on the read data information to obtain a verification result; and if the verification result is successful, report the data information to the data server.

[0127] The aforementioned data processing system deploys a separate data reporting interface configured with data reporting conditions within the data center equipment. Data reporting is achieved through a data proxy process interconnected with the data server. This means each data reporting interface only needs to handle its own local data reporting, reducing the load on individual interfaces and improving efficiency. Furthermore, the data center equipment is configured with shared storage space shared by the business processing process and the data proxy process, allowing for the simultaneous writing and reading of data, further improving efficiency. By calling the data proxy process to read data from the shared storage space according to the time period matched by the business processing process, batch processing of data information can be achieved while ensuring real-time data reporting, further enhancing efficiency. Therefore, the data processing method of this application can improve efficiency during the process of data center equipment reporting data information to the data server.

[0128] In one embodiment, such as Figure 6As shown, the data processing system also includes a proxy server 502; the data center device 501 specifically reports at least one piece of data information to the data server 503 through the proxy server 502; there are multiple data center devices 501 and multiple proxy servers 502. Each data proxy process in each data center device 501 establishes a many-to-one long connection relationship with each proxy server 502.

[0129] Long-lived connections are a technique for establishing persistent communication channels between clients and servers. Once established, this connection persists until explicitly closed or disconnected due to network issues. Compared to traditional short-lived connections (which require re-establishing a connection for each request), long-lived connections reduce the overhead of connection establishment and termination, thus improving data transmission efficiency.

[0130] In practical applications, one data proxy process is deployed on each IDC device. Each data proxy process connects to a proxy server and uploads data collected from the IDC device to the data server through the proxy server. Each data proxy process connects to a unique proxy server, and the number of data proxy processes connected to the same proxy server can be multiple or a single one. For example, ... Figure 7 As shown, data proxy process 1, data proxy process 2, and data proxy process 3 are connected to the same proxy server, namely proxy server 1.

[0131] In the above embodiments, the data proxy process and the proxy server adopt a many-to-one long connection relationship, which not only ensures that the same IDC device uploads data to the data server through a unique proxy server, which is beneficial to improving the security of the data transmission link, but also avoids the overhead caused by frequently establishing and disconnecting network connections, which is beneficial to improving data transmission efficiency and enhancing system performance.

[0132] In one embodiment, there are multiple data servers. In this embodiment, the proxy server is specifically used to: parse each received data message to obtain an information identifier carried by the data message; perform data routing according to the information identifier to determine a target server matching the information identifier from among the data servers; and report the data message to the target server.

[0133] The information identifier may include at least one of numbers, symbols, or letters. Optionally, the information identifier may be determined by the data center equipment. Optionally, when the data type is business interaction data, the data center equipment may determine the information identifier carried by the data information based on the business party interacting with the data center equipment; when the data type is hardware performance data, the data center equipment may determine the equipment identifier of the data center equipment as the information identifier carried by the data information. It is understood that data information carrying the same information identifier is more likely to be related.

[0134] Specifically, for each received data message, the proxy server parses the data message to obtain the information identifier carried by that data message. In one optional embodiment, the data proxy process sends data messages to the proxy server via data packets, and the proxy server can extract the information identifier carried by the data message by parsing the data packets. In another optional embodiment, the proxy server can parse the data message by traversing the number of tags or by direct search to determine the information identifier carried by the data message. In yet another optional embodiment, the proxy server can parse the data message according to its format to determine the information identifier carried by the data message. Once the information identifier is determined, the proxy server can route the data according to the information identifier, determine the target server matching the information identifier from among the data servers, and report the data message to the target server.

[0135] In an optional embodiment, the proxy server can perform hash calculations using the addresses or hostnames of each data server as keys, according to a pre-defined hash algorithm or a consistent hash algorithm, mapping each data server to a hash ring. Upon receiving data information, the proxy server can calculate the information identifier of the data information, map the data information to the hash ring, and further determine the data server corresponding to the mapped position as the target server.

[0136] In an optional embodiment, the proxy server can adjust the hash algorithm according to load balancing requirements and data distribution to achieve load balancing while ensuring that data information corresponding to the same information identifier is reported to the same data server. For example, given a determined mapping range for each data server on the hash ring, the hash algorithm can be adjusted based on the numerical range of the information identifier to ensure that the hash calculation results matching that numerical range cover each mapping range as evenly as possible, thus achieving load balancing. For instance, if the hash values ​​of three data servers are 150, 250, and 350 respectively, then their respective mapping ranges on the hash ring are (0, 150], (150, 250], and (250, 350). If, based on the numerical range of the information identifier, the set hash algorithm predicts that the hash value of the information identifier is likely to be concentrated below 150, then the data server with a hash value of 150 may be overloaded. In this case, load balancing can be achieved by adjusting the hash algorithm so that the hash value of the information identifier covers the numerical range of (0, 350).

[0137] In the above embodiments, data routing is performed according to the information identifier carried by the data information. The target server that matches the information identifier is determined from each data server, and the data information is reported to the target server. This ensures that the associated data is uploaded to the same data server, which facilitates subsequent centralized processing and helps to further improve work efficiency.

[0138] In one embodiment, the data server is specifically used to: cache the data information reported by the data center equipment to the data storage space; when the cached data in the data storage space meets the aggregation conditions, perform aggregation processing on the cached data to obtain aggregated data; and write the aggregated data to the database in batches.

[0139] Data caching refers to temporarily storing data on high-speed storage media to improve data access speed. Common caching technologies include memory caching (such as Redis) and local file caching. Databases can be integrated into data servers, set up independently, or integrated into other servers or devices. Specifically, the database can be a relational database (such as MySQL, PostgreSQL) or a NoSQL database (such as MongoDB, Cassandra).

[0140] Specifically, after receiving data from the data center equipment, the data server first caches the data in its data storage space. This data storage space can be the data server's local storage or external storage; no specific limitation is made here. The data server can periodically check the cached data in the data storage space and, based on preset aggregation conditions, decide whether to aggregate the cached data and write it to the database. If the cached data in the data storage space does not meet the aggregation conditions, the process of acquiring and caching data continues. If the cached data in the data storage space meets the aggregation conditions, the cached data is aggregated to obtain aggregated data, which is then written to the database in batches.

[0141] There is no single way to aggregate cached data. A data server can aggregate all cached data in its data storage space into a single aggregated dataset, or it can aggregate cached data into multiple aggregated datasets. Optionally, the data server can also aggregate data based on its generation time, focusing on data whose generation time difference is less than or equal to a time difference threshold. Optionally, the data server can determine the information identifier carried by each piece of data in the cached data; for each information identifier, it can aggregate the data carrying that identifier in the cached data to obtain the aggregated data corresponding to that identifier.

[0142] In an alternative embodiment, the data server can use batch insert or batch update operations to write aggregated data to the database to improve write efficiency.

[0143] In an optional embodiment, the data server can employ a transaction mechanism during data aggregation and writing to ensure data consistency. The transaction mechanism ensures that if any error occurs during data writing, all operations can be rolled back, preventing data inconsistency.

[0144] In an optional embodiment, the data server can use data partitioning technology to write aggregated data to multiple partitions of the database to improve the database's read and write performance.

[0145] In an optional embodiment, the data server can create indexes for frequently used query fields and maintain the indexes regularly to avoid index fragmentation affecting performance and improve data query efficiency.

[0146] In an optional embodiment, the database can be backed up periodically. The backup strategy can be configured based on the importance and frequency of data changes to ensure rapid data recovery in the event of data loss or corruption, thus guaranteeing normal system operation.

[0147] In the above embodiments, data information is first cached. When the cached data in the data storage space meets the aggregation conditions, the cached data is then aggregated to obtain aggregated data, which is then written to the database. This can significantly reduce the number of database writes, reduce the database write pressure, and further improve work efficiency.

[0148] In an optional embodiment, the cached data is aggregated to obtain aggregated data, including: determining the information identifier carried by each data information contained in the cached data; and for each information identifier, aggregating the data information carrying that information identifier in the cached data to obtain the aggregated data corresponding to the information identifier.

[0149] The specific limitations regarding information identifiers are detailed above and will not be repeated here. Specifically, data carrying the same information identifier are highly likely to be related data. For example, hardware performance data collected from the same data center equipment carries the same information identifier; similarly, log data generated by the same interacting party during interactions with the data center equipment also carries the same information identifier. Based on this, the data server can parse the cached data to determine the information identifier carried by each piece of data within the cached data. Then, for each information identifier, the data information carrying that information identifier in the cached data is aggregated to obtain the aggregated data corresponding to the information identifier, and finally, the aggregated data is written to the database.

[0150] In the above embodiments, data aggregation based on the information identifiers carried by the data information can ensure that related data is written to the database in batches, which facilitates subsequent centralized processing and helps to further improve work efficiency.

[0151] Furthermore, the aggregation condition can be characterized by at least one of the data volume or the caching duration. In an optional embodiment, the data server is also configured to perform at least one of the following two actions: determining that the cached data satisfies the aggregation condition when the data volume of the cached data reaches a data volume threshold; and determining that the cached data satisfies the aggregation condition when the caching duration of the cached data reaches a duration threshold.

[0152] Specifically, cached data can include multiple data entries, and the amount of cached data refers to the total amount of these data entries. Since the cache duration for each data entry may be different, the cache duration of cached data can refer to the cumulative cache duration, average cache duration, or maximum cache duration of all data entries.

[0153] Specifically, the data server can determine that cached data meets the aggregation conditions when the amount of cached data reaches a data volume threshold and / or the cache duration reaches a duration threshold. Optionally, the data server can dynamically adjust the aggregation conditions to adapt to different business needs. For example, in troubleshooting or maintaining equipment in an IDC environment, high data timeliness is usually required. In this case, the duration threshold or data volume threshold can be shortened to ensure that cached data is written to the database in a timely manner. Conversely, when the data server or database is under heavy load, the duration threshold or data volume threshold can be extended to slow down data aggregation and writing frequency, thereby reducing the load.

[0154] In the above embodiments, whether the aggregation condition is met is determined based on at least one of the amount of cached data or the cache duration. The data amount threshold or the duration threshold can be adjusted according to specific needs to achieve flexible configuration of the aggregation condition, which is conducive to improving the scientific nature of the data processing process.

[0155] The following is combined with Figure 8 and Figure 9 This paper provides a detailed description of the data processing procedure in this application.

[0156] In one specific embodiment, the data processing involves IDC equipment, a proxy server (DataProxy), a data server (DataSvr), and a database. For example... Figure 8 As shown, the IDC device deploys business processing processes, data agent processes, and a shared storage space shared by the business processing processes and data agent processes. This shared storage space can be shared memory.

[0157] The business processing process is a core component deployed on the IDC (Internet Data Center) equipment, responsible for executing specific business logic. Because it handles a large amount of computation and data operations, it consumes a significant portion of the IDC equipment's resources, generating data that needs to be reported. To fully understand the IDC equipment's operational status, the business processing process can collect data from it. When data needs to be reported, the business processing process can call a dedicated data reporting interface. This interface can be a predefined one, ensuring that data is correctly written to the shared storage space according to the agreed-upon format and data range. Optionally, containerization technology (such as Docker) can be used for the deployment of the business processing process for easier management and expansion. Optionally, the reported data format can use standardized formats such as JSON, XML, or Protobuf to ensure readability and parsability. Optionally, a robust error handling mechanism can be configured for the business processing process to ensure retries or logging in case of data reporting failures for subsequent analysis and processing.

[0158] Shared storage space, which can be shared memory, is a dedicated space on the IDC device used to store data reported by business processing processes. The shared storage space is connected to both the business processes and the data agent. The business processes are responsible for writing data to the shared storage space, while the data agent is responsible for reading and processing data from it. Optionally, the shared storage space can use a ring buffer or linked list structure to improve read / write efficiency. Furthermore, to avoid data read / write conflicts, synchronization mechanisms (such as semaphores or mutexes) can be used to ensure data consistency and integrity.

[0159] DataAgent is a process deployed on the IDC device responsible for reading and processing data from shared storage. Optionally, a timer can be configured to match DataAgent, causing it to periodically attempt to retrieve data written by the business processing process. If valid data is found, DataAgent reads it, performs basic processing such as data validation and format conversion, and then sends it to DataProxy via a network connection. Optionally, the timer interval can be configured according to business needs, ensuring data timeliness while avoiding the performance overhead of frequent reads. Optional, such as... Figure 9 As shown, a many-to-one long-lived connection relationship is used between DataAgent and DataProxy. One DataAgent will only connect to one DataProxy, which can avoid the overhead caused by frequently establishing and disconnecting network connections and is beneficial to improving system performance.

[0160] DataProxy is responsible for connecting to multiple DataAgents and receiving data sent from them. Upon receiving data, DataProxy first parses the data to extract its identifier (data ID). Then, DataProxy calculates the data ID using a pre-defined hash algorithm or consistent hashing algorithm to determine the target DataSvr to which the data should be routed. For example, ... Figure 9 As shown, data with a hash calculation result of 1 (i.e., uin=1) will be routed to DataSvr_1, data with a hash calculation result of 2 (i.e., uin=2) will be routed to DataSvr_2, and data with a hash calculation result of 3 (i.e., uin=3) will be routed to DataSvr_3. That is, there is a many-to-many connection between DataProxy and DataSvr. The choice of hash algorithm can be adjusted according to the system's load balancing requirements and data distribution. Routing based on data ID ensures that data with the same ID is sent to the same DataSvr, avoiding the processing complexity caused by data dispersion.

[0161] After receiving data from DataProxy, DataSvr first caches the data locally. This caching can use in-memory caching (such as Redis) or local file caching. The cache size and duration are configured according to business needs to ensure that the cache does not consume excessive resources while effectively storing high-frequency data over a short period. DataSvr periodically checks the data in the cache and, based on preset cache time and size policies, decides whether to aggregate and write the cached data to the database. The cache management strategy (corresponding to the aggregation conditions mentioned earlier) can be dynamically adjusted according to business needs to adapt to different load conditions. After the cache management strategy is triggered, DataSvr aggregates the data in the cache. Aggregation is performed by merging data with the same data ID together. After data aggregation is complete, DataSvr writes the aggregated data to the database in batches. Batch writing significantly reduces the number of database writes and lowers the database write pressure. Optionally, DataSvr can use batch insert or batch update operations to improve write efficiency. Optionally, DataSvr can employ transaction mechanisms during data aggregation and writing to ensure data consistency. Transaction mechanisms ensure that if any error occurs during the data writing process, all operations can be rolled back, preventing data inconsistency.

[0162] The database is responsible for storing data sent from DataProxy. A suitable database can be chosen to meet business needs, such as a relational database (e.g., MySQL, PostgreSQL) or a NoSQL database (e.g., MongoDB, Cassandra). Optionally, data partitioning can be used to distribute data across multiple partitions for storage, improving database read and write performance. Optionally, indexes can be created on frequently queried fields and maintained regularly to avoid index fragmentation impacting performance, thus improving data query efficiency. Optionally, the database can be backed up regularly, combined with a data recovery strategy, to address data loss or corruption. The backup strategy can be configured based on the importance and frequency of data changes. The data recovery strategy ensures rapid data recovery in the event of data loss or corruption, guaranteeing the normal operation of the system.

[0163] In the above embodiments, the efficient collection and reporting of large-scale data in an IDC environment can be achieved through the collaborative work of multiple components such as business processing processes, shared storage space, DataAgent, DataProxy, DataSvr, and database. Furthermore, each component is designed with high concurrency, high reliability, and high performance in mind to ensure stable system operation and accurate data transmission. The entire process, from data generation to final database writing, undergoes multi-level processing and optimization, ensuring efficient data transmission and storage.

[0164] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0165] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0166] In one embodiment, such as Figure 10As shown, a data processing device is provided, applied to data center equipment; the data center equipment deploys a business processing process, a data reporting interface, and a data proxy process. In this embodiment, the data processing device includes a data acquisition module 1001, a shared storage module 1002, a data reading module 1003, and a data reporting module 1004. Wherein:

[0167] The data acquisition module 1001 is used to respond to the call event of the business processing process to the data reporting interface, and to acquire the data information reported by the business processing process through the data reporting interface; the data information meets the data reporting conditions configured for the data reporting interface.

[0168] The shared storage module 1002 is used to write data information into the shared storage space; the shared storage space is a storage space shared by the business processing process and the data agent process.

[0169] The data reading module 1003 is used to call the data agent process according to the time period matched by the business processing process to read at least one piece of data information written within the time period from the shared storage space;

[0170] The data reporting module 1004 is used to report at least one piece of data information to the data server.

[0171] In one embodiment, the data reporting conditions are characterized by data range and data format; the data processing device further includes a call event triggering module, used to: when the original data generated during the business processing is obtained, filter selected data that meets the data range from the original data; and when the selected data is format-converted to obtain data information that conforms to the data format, trigger the call event of the business processing process for the data reporting interface.

[0172] In one embodiment, the data processing apparatus further includes: a lock adding module, configured to add a mutex lock to the data information when there is an idle mutex lock, and change the state of the mutex lock to a locked state; and a lock releasing module, configured to release the mutex lock and change the state of the mutex lock to an idle state when the data information is written to the shared storage space.

[0173] In one embodiment, the data reading module 1003 is specifically used to: determine a timer configured for the business processing process; trigger the timer according to a time period matching the business processing process; and, when the timer is triggered, call the data agent process to read at least one piece of data information written within the time period from the shared storage space.

[0174] In one embodiment, the data reporting module 1004 is specifically used to: perform data verification on the read data information and obtain the verification result; if the verification result is that the verification is successful, report the data information to the data server through the proxy server.

[0175] In one embodiment, there are multiple data servers; the proxy server specifically routes data according to the information identifier carried by each data message, and reports each data message to the corresponding data server. In this embodiment, the data processing device further includes an information identifier determination module, used to: determine the data type of the data message; if the data type is business interaction data, determine the information identifier carried by the data message based on the business party interacting with the data center equipment; if the data type is hardware performance data, determine the device identifier of the data center equipment as the information identifier carried by the data message.

[0176] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0177] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data involved in the aforementioned data processing. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0178] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0179] Those skilled in the art will understand that Figure 11 or Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method embodiment described above.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method embodiment described above.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method embodiments described above.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions and areas. Furthermore, the recipient may choose not to authorize the use of their information and related data, or may refuse or conveniently refuse to receive push notifications.

[0184] In this application, during the actual application of relevant data collection and processing, the informed consent or separate consent of the personal information subject should be obtained in strict accordance with the requirements of relevant local laws and regulations, and subsequent data use and processing should be carried out within the scope of laws and regulations and the authorization of the personal information subject.

[0185] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method is applied to data center equipment; the data center equipment is deployed with business processing processes, data reporting interfaces, and data proxy processes; the method includes: In response to the business processing process's call event to the data reporting interface, the system obtains the data information reported by the business processing process through the data reporting interface; the data information satisfies the data reporting conditions configured for the data reporting interface. The data information is written into a shared storage space; the shared storage space is a storage space shared by the business processing process and the data proxy process. According to the time period matched by the business processing process, the data proxy process is invoked to read at least one piece of data information written within the time period from the shared storage space; The at least one piece of data information shall be reported to the data server.

2. The method according to claim 1, characterized in that, The data reporting conditions are characterized by data range and data format; the method further includes: When the raw data generated during the business process is obtained, selected data that meets the data range is filtered from the raw data; When the selected data is converted to a format that conforms to the data format, a call event is triggered in the business processing process for the data reporting interface.

3. The method according to claim 1, characterized in that, The method further includes: If an idle mutex exists, add the mutex to the data information and change the state of the mutex to locked. When the data information is written to the shared storage space, the mutex is released and the state of the mutex is changed to an idle state.

4. The method according to claim 1, characterized in that, The step of invoking the data proxy process according to the time period matched by the business processing process, and reading at least one piece of data information written within the time period from the shared storage space, includes: A timer configured for the business processing process is determined; the timer is triggered according to a time period matching the business processing process. When the timer is triggered, the data agent process is invoked to read at least one piece of data information written within the time period from the shared storage space.

5. The method according to any one of claims 1 to 4, characterized in that, The step of reporting the at least one piece of data information to the data server includes: The read data information is validated to obtain the validation result; If the verification result is successful, the data information is reported to the data server through the proxy server.

6. The method according to claim 5, characterized in that, The number of data servers is multiple; the proxy server specifically performs data routing according to the information identifier carried by each data information, and reports each data information to the corresponding data server respectively. The method further includes: Determine the data type of the data information; When the data type is business interaction data, the information identifier carried by the data information is determined based on the business party that interacts with the data center equipment. When the data type is hardware performance data, the device identifier of the data center device is determined as the information identifier carried by the data information.

7. A data processing system, characterized in that, This includes data center equipment and data servers; the data center equipment is equipped with business processing processes, data reporting interfaces, and data proxy processes. The data center equipment is configured to respond to a call event from the business processing process to the data reporting interface, obtain data information reported by the business processing process through the data reporting interface; write the data information into a shared storage space; call the data proxy process according to a time period matched by the business processing process, and read at least one piece of data information written within the time period from the shared storage space; report the at least one piece of data information to the data server; the data information satisfies the data reporting conditions configured for the data reporting interface; the shared storage space is a storage space shared by the business processing process and the data proxy process. The data server is used to store the data information reported by the data center equipment.

8. The system according to claim 7, characterized in that, The system also includes a proxy server; the data center equipment specifically reports the at least one piece of data information to the data server through the proxy server. There are multiple data center devices and multiple proxy servers; each data proxy process in each data center device establishes a many-to-one long connection relationship with each proxy server.

9. The system according to claim 8, characterized in that, The number of data servers is multiple; the proxy server is specifically used for: For each received data information, the data information is parsed to obtain the information identifier carried by the data information; Data routing is performed according to the information identifier, and a target server matching the information identifier is determined from each of the data servers; The data information is reported to the target server.

10. The system according to claim 7, characterized in that, The data server is specifically used for: The data information reported by the data center equipment is cached in the data storage space; If the cached data in the data storage space meets the aggregation conditions, the cached data is aggregated to obtain aggregated data. The aggregated data is written to the database in batches.

11. The system according to claim 10, characterized in that, The aggregation process of the cached data to obtain aggregated data includes: Determine the information identifier carried by each piece of data contained in the cached data; For each of the aforementioned information identifiers, the data information carrying the information identifier in the cached data is aggregated to obtain the aggregated data corresponding to the information identifier.

12. The system according to claim 10, characterized in that, The data server is also configured to perform at least one of the following two: If the amount of cached data reaches a data volume threshold, it is determined that the cached data meets the aggregation condition. If the caching duration of the cached data reaches a duration threshold, the cached data is determined to meet the aggregation condition.

13. A data processing apparatus, characterized in that, Applied to data center equipment; the data center equipment is equipped with service processing processes, data reporting interfaces, and data proxy processes; the device includes: The data acquisition module is used to respond to the call event of the business processing process to the data reporting interface, and acquire the data information reported by the business processing process through the data reporting interface; the data information satisfies the data reporting conditions configured for the data reporting interface. A shared storage module is used to write the data information into a shared storage space; the shared storage space is a storage space shared by the business processing process and the data proxy process. The data reading module is used to call the data agent process according to the time period matched by the business processing process, and read at least one piece of data information written within the time period from the shared storage space; The data reporting module is used to report the at least one piece of data information to the data server.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images