A data collection method and related system

Abstract

The present application relates to a data collection method and system. The method is applied to any service node in a server cluster, and the service node includes a first module and a second module; the method includes: receiving a report request forwarded by a load balancing server through the first module, storing the tracking data carried by the report request in the local file system, and returning a response message indicating a successful report to the user terminal that sent the report request; asynchronously sending the tracking data stored in the local file system to a message queue through the second module. The embodiment of the present application can improve the processing performance of the server in scenarios with high concurrent traffic and provide users with a better user experience.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and at least to a data collection method and related system. Background Art

[0002] The following statements merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Currently, to provide higher-quality services, application providers collect user data (with user permission) and analyze it to adjust service details. A common data collection method involves using tracking technology to have the application client report user-related tracking data, such as user login behavior, to the application server, which then analyzes the data reported by the client.

[0004] In some scenarios, the reporting process of tracking data is as follows: the client first establishes a connection with the server, and then reports the tracking data through the connection. When the server receives the tracking data, it sends it to the message queue for consumers (such as data analysis systems) to consume. After the server sends the tracking data to the message queue, it returns relevant information to the client to inform the client that the tracking data has been successfully reported and disconnects.

[0005] The inventors noticed that in the above-mentioned reporting process of the buried data, after the server receives the buried data, it needs to send the buried data to the message queue before responding to the client and disconnecting the connection. In scenarios with low concurrency and low traffic, the above-mentioned reporting process usually does not have adverse effects. However, in some scenarios with high concurrency and large traffic, the server needs to connect to a large number of clients. At this time, for the server, if it maintains a connection with a large number of clients for a long time, it will occupy a lot of computing resources, resulting in low processing performance, and may also cause an inability to respond to other clients in a timely manner; for the client, maintaining a connection with the server for a long time will also occupy the computing resources of the user terminal where the client is located, thereby affecting the user experience. Summary of the Invention

[0006] In response to the above-mentioned deficiencies or shortcomings, the present application provides a data collection method, service node, data collection system, computer equipment and storage medium. The embodiments of the present application can improve the processing performance of the server in scenarios with high concurrent traffic and provide users with a better user experience.

[0007] According to a first aspect, the present application provides a data collection method. In some embodiments, the method is applied to any service node in a server cluster, where the service node includes a first module and a second module. The method includes:

[0008] The first module receives the report request forwarded by the load balancing server, stores the tracking data carried by the report request in the local file system, and returns a response message indicating that the report is successful to the user terminal that sent the report request;

[0009] The second module asynchronously sends the tracking data stored in the local file system to the message queue.

[0010] In some embodiments, receiving a report request forwarded by a load balancing server through a first module and storing the tracking data carried in the report request into a local file system includes:

[0011] Create multiple processes and multiple disk flushing task queues corresponding to the multiple processes;

[0012] Generate a corresponding disk placement task according to each report request forwarded by the load balancing server, and add the generated disk placement task to one of the multiple disk placement task queues;

[0013] Each process obtains the disk placement task from the corresponding disk placement task queue, and writes the tracking data carried by the reporting request indicated by the disk placement task into the local file system.

[0014] In some embodiments, the number of user terminals is multiple; the load balancing server provides multiple reporting interfaces for each user terminal, and each user terminal sends a reporting request carrying burial point data to the load balancing server through a target reporting interface among the multiple reporting interfaces, and the target reporting interface refers to a reporting interface corresponding to the business type of the burial point data carried by the reporting request;

[0015] A first directory of the local file system is provided with a plurality of data files;

[0016] When the target process writes the tracking data carried in the target reporting request to the local file system, it includes:

[0017] Obtaining, through the target process, an interface identifier of a reporting interface for sending a target reporting request; wherein the target process refers to any process, and the target reporting request refers to any reporting request;

[0018] According to the preset first corresponding relationship, the target write data file corresponding to the interface identifier in the above-mentioned multiple data files is determined, and the burial point data carried by the target reporting request is written into the target write data file; the preset first corresponding relationship includes the corresponding relationship between the above-mentioned multiple reporting interfaces and the above-mentioned multiple data files.

[0019] In some embodiments, when the target process writes the tracking data carried by the target reporting request into the target write data file, it includes:

[0020] Query the target process to see if there is a handle to the target write data file.

[0021] If so, write the tracking data carried by the target reporting request into the target writing data file according to the handle of the target writing data file;

[0022] If not, request the local file system to open the target write data file to obtain the handle of the target write data file, save the obtained handle, and write the burial point data carried by the target reporting request into the target write data file according to the obtained handle.

[0023] In some embodiments, each data file in the first directory is provided with a corresponding file lock;

[0024] Before writing the tracking data carried in the target reporting request into the target data file according to the handle of the target writing data file, the following is also included:

[0025] The target process competes with other processes for the target file lock, which refers to the file lock corresponding to the target write data file;

[0026] If the target file lock is obtained, proceed to the next step;

[0027] If the target file lock cannot be obtained, wait.

[0028] In some embodiments, asynchronously sending the tracking data stored in the local file system to the message queue through the second module includes:

[0029] Create multiple threads through the second module;

[0030] The embedded point data in the above multiple data files are sent to the message queue through the above multiple threads.

[0031] In some embodiments, a message queue is provided with multiple topics; when the embedded data in the target data file is sent to the message queue through the target thread, it includes:

[0032] Obtain the file identifier of the target data file through the target thread; the target thread refers to any thread, and the target data file refers to any data file in the first directory;

[0033] According to the preset second corresponding relationship, the target topic corresponding to the file identifier among the above-mentioned multiple topics is determined, and the buried point data in the target data file is sent to the target topic; the preset second corresponding relationship includes the corresponding relationship between the above-mentioned multiple data files and the above-mentioned multiple topics.

[0034] In some embodiments, the above method further comprises:

[0035] A first module is used to periodically create a file with the same name as a corresponding replacement file for each original data file in the first directory, and each original data file is transferred to a second directory in the local file system.

[0036] In some embodiments, the above method further comprises:

[0037] In the process of transferring any original data file in the first directory to the second directory, the burial point data carried by each received reporting request associated with any data file is written into any data file until the transfer process is completed.

[0038] According to a second aspect, the present application provides a service node. In some embodiments, the service node includes a first module and a second module;

[0039] The first module is used to receive the report request forwarded by the load balancing server, store the tracking data carried in the report request in the local file system, and return a response message indicating that the report is successful to the user terminal that sent the report request;

[0040] The second module is used to asynchronously send the tracking data stored in the local file system to the message queue.

[0041] According to a third aspect, the present application provides a data acquisition system, comprising a load balancing server, a server cluster consisting of multiple service nodes, and a message queue; each service node is provided with a first module and a second module; wherein,

[0042] The load balancing server is used to receive the report request sent by the user terminal and forward the report request to the service node in the server cluster;

[0043] Each service node is used to receive the reporting request forwarded by the load balancing server through the first module, store the tracking data carried in the reporting request in the local file system, return a response message indicating that the report is successful to the user terminal that sent the reporting request, and asynchronously send the tracking data stored in the local file system to the message queue through the second module.

[0044] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data acquisition method provided in any of the above embodiments are implemented.

[0045] According to a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the data acquisition method provided in any of the above embodiments are implemented.

[0046] In the above embodiment of the present application, after the service node extracts the buried point data, it will not directly send the buried point data to the message queue. The processing of the reporting request by the service node can be regarded as two parts, namely "data storage" and "data transmission", which are respectively executed by the first module and the second module in the service node, and the processing logic of the first module and the second module are asynchronous, and the two are independent of each other. Among them, the processing process of "data storage" is: the first module receives the reporting request forwarded by the load balancing server, and stores the buried point data carried by each received reporting request into the local file system. After storing the buried point data carried by any reporting request into the local file system, a response message indicating that the report is successful is returned to the user terminal that sent the reporting request, so that the connection between the client and the server can be disconnected faster, and the first module can provide services to other clients more promptly. The processing process of "data transmission" is: the second module extracts the buried point data from the local file system, and then, as the producer of the message queue, transfers the extracted buried point data to the message queue.

[0047] Through the above-mentioned embodiments of the present application, in some scenarios with high concurrent traffic, for the server side, the server side can reduce the length of time it maintains connection with the client side when docking with a large number of clients, which can improve the utilization rate of computing resources, provide services to more clients in a unit of time, and improve processing performance; for the client side, it can complete the reporting process of the buried data more quickly, avoiding occupying the computing resources of the user terminal where the client is located for a long time due to reporting the buried data, thereby affecting the user experience. Furthermore, this embodiment persists the buried data to a data file (there can be multiple) in the local file system, so that even if the service node fails or even the message queue fails, the buried data reported by the user terminal will not be lost, which can ensure data security and improve availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is an application environment diagram of a data collection method provided by the present application according to one or more embodiments;

[0049] Figure 2 A flowchart of a data collection method provided by the present application according to one or more embodiments;

[0050] Figure 3 A schematic diagram of a process for processing a report request through multiple processes according to one or more embodiments of the present application;

[0051] Figure 4 A schematic diagram of a process for writing buried data to a target data file according to one or more embodiments of the present application;

[0052] Figure 5A schematic diagram of a process for sending tracking data to a target topic according to one or more embodiments of the present application;

[0053] Figure 6 This is a structural block diagram of a service node provided by the present application according to one or more embodiments;

[0054] Figure 7 This is a diagram of the internal structure of a computer device provided in accordance with one or more embodiments of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] The present application provides a data collection method. In some embodiments, the data collection method can be applied to Figure 1 In the application environment shown. Among them, the server involved in the embodiment of the present application includes at least a load balancing server, a server cluster and a message queue, wherein the server cluster contains multiple service nodes, the processing logic of each service node is consistent, and the number of service nodes can be dynamically expanded or reduced, and the message queue can be a middleware such as kafka. The client of one or more applications will report the buried data to the server. Among them, the client is installed in the user terminal, and the user terminal may include but is not limited to various personal computers, laptops, smart phones, tablet computers and desktop computers.

[0057] A data collection method provided in this embodiment includes the following steps: Figure 2 The steps shown are described below using the method applied to any service node in a server cluster as an example.

[0058] S110: Receive, through the first module, a report request forwarded by the load balancing server, store the tracking data carried in the report request in the local file system, and return a response message indicating a successful report to the user terminal that sent the report request;

[0059] S120: Asynchronously send the tracking data stored in the local file system to the message queue through the second module.

[0060] In this embodiment, the user terminal will report the generated tracking data to the server, and the load balancing server in the server will forward the received reporting request according to the preset forwarding rules (for example, it can be forwarded by reverse proxy) to the service node in the server cluster for processing. This embodiment does not specifically limit the above forwarding rules, as long as the purpose of load balancing can be achieved. For example, when forwarding the reporting request, the reporting request will be forwarded to the service node with the least number of pending reporting requests in the server cluster. There are multiple user terminals mentioned above, and the processing logic of each user terminal is consistent. The user terminal has installed the client of the application, and the client can integrate a plug-in or SDK (Software Development Kit) for reporting tracking data. When the user uses the client through the user terminal, the plug-in or SDK will carry the generated tracking data in the reporting request and send it to the server. This embodiment does not specifically limit the communication protocol used by the client to send the reporting request. For example, it can be Hyper Text Transfer Protocol (HTTP).

[0061] After each service node in the server cluster receives the report request forwarded by the load balancing server, it needs to extract the tracking data carried in the report request and then transmit it to the message queue for consumption by the consumers of the message queue. It should be noted that after the service node extracts the tracking data, it will not directly send the tracking data to the message queue. The service node's processing of the report request can be regarded as two parts, namely "data storage" and "data transmission", which are executed by the first module and the second module in the service node respectively. The processing logic of the first module and the second module is asynchronous and independent of each other. Among them, the processing process of "data storage" is as follows: the first module receives the report request forwarded by the load balancing server and stores the tracking data carried by each received report request in the local file system. After storing the tracking data carried by any report request in the local file system, it returns a response message indicating that the report is successful to the user terminal that sent the report request, so that the connection between the client and the server can be disconnected faster, and the first module can provide services to other clients more promptly. The processing process of "data transmission" is as follows: the second module extracts the tracking data from the local file system, and then, as the producer of the message queue, transmits the extracted tracking data to the message queue. Optionally, in some scenarios, the reporting request sent by the user terminal will first come to the reverse proxy server. The reverse proxy server is responsible for interacting with the user terminal (such as receiving requests, returning the processing results of the requests, etc.). The reverse proxy server sends the reporting request to the load balancing server, which then forwards it to the relevant service node in the server cluster for processing. In this scenario, the service node can return the response information to the relevant user terminal through the reverse proxy server.

[0062] Through the above embodiments, in some scenarios with high concurrent traffic, for the server side, when connecting to a large number of clients, the server side can reduce the length of time it maintains a connection with the client side, which can improve the utilization rate of computing resources, provide services to more clients per unit time, and improve processing performance; for the client side, it can complete the reporting process of the buried data more quickly, avoiding occupying the computing resources of the user terminal where the client is located for a long time due to reporting the buried data, thereby affecting the user experience. Furthermore, this embodiment persists the buried data to a data file (there can be multiple) in the local file system, so that even if the service node fails or even the message queue fails, the buried data reported by the user terminal will not be lost, which can ensure data security and improve availability.

[0063] Furthermore, in some embodiments, the first module receives the report request forwarded by the load balancing server and stores the buried data carried in the report request in the local file system, such as Figure 3 As shown, steps S111-S113 are included.

[0064] S111: Create multiple processes and multiple disk flushing task queues corresponding to the multiple processes;

[0065] S112: Generate a corresponding disk placement task according to each report request forwarded by the load balancing server, and add the generated disk placement task to one of the multiple disk placement task queues;

[0066] S113: Obtain the disk placement task from the corresponding disk placement task queue through each process, and write the tracking data carried by the reporting request indicated by the disk placement task into the local file system.

[0067] In this embodiment, in order to cope with high concurrency and large traffic, the first module of each service node processes the reporting request forwarded by the load balancing server through multiple processes to achieve efficient reception of the buried data reported by the user terminal. Among them, the first module will create multiple processes, and each process is used to implement the above-mentioned "data drop to disk" processing process. Specifically, the first module will create a corresponding task queue for each process (referred to as a drop to disk task queue in this embodiment). After the first module receives each reporting request forwarded by the load balancing server, it will generate a drop to disk task and add it to the drop to disk task queue corresponding to one of the processes. Each process executes the drop to disk task in the corresponding drop to disk task queue in a first-in-first-out manner. Optionally, in order to make more full use of computing resources, a scheduling process can be created. The scheduling process can record the length of the drop to disk task queue corresponding to each of the above processes, and when receiving a reporting request, it will decide which process's drop to disk task queue to add the reporting request to based on the length of the drop to disk task queue corresponding to each of the above processes. For example, the reporting request can be added to the drop to disk task queue with the shortest length.

[0068] Optionally, the first module can be implemented using openresty (OpenResty is a high-performance web platform based on Nginx and Lua, which integrates a large number of sophisticated Lua libraries, third-party modules, and most dependencies).

[0069] The following describes the process of any process writing tracking data to the local file system. In some embodiments, any of the above processes used to implement the "data storage" process is referred to as the target process, and any report request forwarded by a load balancing server to any service node is referred to as the target report request; at this time, the target process writes the tracking data carried by the target report request to the local file system, such as Figure 4 Shown, including:

[0070] S210: Obtaining, through the target process, an interface identifier of a reporting interface for sending a target reporting request.

[0071] S220: Determine the target write data file corresponding to the interface identifier in the above-mentioned multiple data files according to the preset first corresponding relationship, and write the burial point data carried by the target reporting request into the target write data file.

[0072] The preset first correspondence includes the correspondence between the multiple reporting interfaces and the multiple data files. The target write data file refers to the write data file corresponding to the interface identifier (the interface identifier of the reporting interface for sending the target reporting request) in the multiple data files.

[0073] In this embodiment, in order to more efficiently receive the buried data reported by the user terminal in a high-concurrency and high-traffic scenario, the load balancing server (or the above-mentioned reverse proxy server) provides multiple reporting interfaces for each user terminal (such as an interface for collecting payment data, an interface for collecting login behavior data, etc.). At the same time, a plurality of data files for storing buried data are preset in the local file system (for the convenience of management, the plurality of data files are stored in the same directory, hereinafter referred to as the first directory). A correspondence is established between the plurality of reporting interfaces and the plurality of data files. For example, a correspondence can be established between the interface identifier of each reporting interface and the file identifier of the data file. By presetting multiple reporting interfaces and multiple data files, one reporting interface corresponds to one data file. When any user terminal sends a reporting request carrying buried data to the load balancing server through the target reporting interface among the above-mentioned multiple reporting interfaces, the first module of the relevant service node can quickly determine the data file corresponding to the target reporting interface through the correspondence and store the buried data in the data file.

[0074] In some scenarios, the correspondence between the multiple reporting interfaces and the multiple data files can be one-to-one, for example, reporting interfaces A, B, and C correspond to data files a, b, and c, respectively. In other possible scenarios, the number of reporting interfaces may be relatively large (for example, hundreds or thousands), and in order to ensure the efficiency of data storage, the number of data files is controlled to a smaller number (for example, dozens). Then each reporting interface uniquely corresponds to one of the data files. It should be understood that at this time, each data file may correspond to one or more reporting interfaces. In addition, in some possible scenarios, the clients of different applications will uniformly report the buried data to the load balancing server (or the above-mentioned reverse proxy server). At this time, when the clients installed by different user terminals correspond to different applications, the reporting interfaces provided by the load balancing server for different user terminals may be partially different or completely different.

[0075] Furthermore, in the above embodiment, when the target process writes the tracking data carried by the target reporting request into the target write data file, the following operations may also be included:

[0076] Query the target process to see if there is a handle to the target write data file.

[0077] If so, write the tracking data carried by the target reporting request into the target writing data file according to the handle of the target writing data file;

[0078] If not, request the local file system to open the target write data file to obtain the handle of the target write data file, save the obtained handle, and write the burial point data carried by the target reporting request into the target write data file according to the obtained handle.

[0079] The inventors realize that in a high-concurrency and high-traffic scenario, each data file needs to be frequently opened and closed. The above-mentioned opening and closing operations take up a lot of overhead in the process of writing the buried data into the data file, thereby affecting the efficiency of receiving the buried data. Therefore, in order to further improve the efficiency of receiving the buried data, this embodiment reuses the handle of each data file. When any process opens any data file, the handle of the data file will be saved, and after completing the buried data writing operation, the data file will not be closed. Therefore, when the target process needs to write buried data to the target write data file, it first executes the operation of querying whether there is a handle of the target write data file locally. If there is a pre-existing handle, the pre-existing handle of the target write data file is directly used to write the buried data to the target write data file, thereby saving the overhead of opening the target write data file. If there is no pre-existing handle, the local file system is requested to open the target write data file. After opening the target write data file, its handle is saved so that other processes can reuse the handle.

[0080] Furthermore, in some embodiments, each data file in the first directory is provided with a corresponding file lock; accordingly, before writing the buried point data carried by the target reporting request into the target write data file according to the handle of the target write data file, the above method also includes: competing with other processes for the target file lock through the target process, the target file lock refers to the file lock corresponding to the target write data file; if the target file lock is obtained, execute the next step; if the target file lock cannot be obtained, wait.

[0081] Since there are multiple processes that need to write buried data to a data file at the same time, in order to avoid the simultaneous writing of data affecting the final writing result, a corresponding file lock can be set for each data file. Before the target process writes data to the target data file, it must first obtain the file lock of the file before executing the subsequent steps, that is, writing data to the file; if the file lock of the file cannot be obtained, it will wait.

[0082] In some embodiments, asynchronously sending the tracking data stored in the local file system to the message queue via the second module may include: creating multiple threads via the second module; and sending the tracking data in the multiple data files to the message queue via the multiple threads. The execution logic of each thread is consistent.

[0083] In the prior art, if the service node waits until the buried data is sent to the message queue after receiving the reporting request before ending the buried data reporting process of the user terminal, then the various operations of this process are performed synchronously, and thus the efficiency of transmitting the buried data to the message queue is relatively low, which is insufficient to cope with high concurrency and large traffic scenarios. However, the above embodiment of the present application performs the above-mentioned "data drop to disk" processing on the buried data through the first module, and then returns relevant response information to the user terminal, thereby ending the buried data reporting process earlier. The first module can serve more user terminals and handle more reporting requests in a unit of time through multiple processes, while the second module can concurrently execute the "data transmission" processing process in a multi-threaded manner in an asynchronous manner, thereby improving the efficiency of transmitting buried data to the message queue.

[0084] The following describes the process of transmitting the buried data from any thread created by the second module to the message queue. In some embodiments, any of the above threads is referred to as the target process, and the data file in any first target is referred to as the target data file. In this embodiment, the message queue has multiple topics. If the message queue is implemented in Kafka, the topic refers to the topic in Kafka. Accordingly, when the buried data in the target data file is sent to the message queue through the target thread, Figure 5 Shown, including:

[0085] S310: Obtain a file identifier of a target data file through a target thread.

[0086] S320: Determine the target topic corresponding to the file identifier among the above multiple topics according to the preset second corresponding relationship, and send the buried point data in the target data file to the target topic.

[0087] The preset second corresponding relationship includes the corresponding relationship between the above-mentioned multiple data files and the above-mentioned multiple topics.

[0088] In this embodiment, in order to further improve the efficiency of transmitting embedded data to the message queue, multiple topics are set in the message queue, and a correspondence is established between the multiple topics and the multiple data files in the first directory (which can be a correspondence between file identifiers and topic identifiers). By presetting multiple topics, one data file corresponds to one topic. When the target thread needs to transmit the embedded data of the target data file to the message queue, the target topic (i.e., the topic corresponding to the target data file) can be quickly determined through the target data file, and the embedded data in the target data file can be transmitted to the target topic.

[0089] In some embodiments, the method further includes: periodically creating, by the first module, a file with the same name as a corresponding replacement file for each original data file in the first directory, and transferring each original data file to a second directory in the local file system.

[0090] In this embodiment, as tracking data is continuously written, the data file will become larger and larger. To prevent the data file from becoming too large and affecting the efficiency of data writing, the service node can periodically rotate the file through the first module. The file rotation interval can be adjusted according to actual needs, for example, 12 hours, 24 hours, etc.

[0091] During file rotation, the first module creates a corresponding file with the same name (i.e., a data file with the same file name) for each data file in the first directory and moves each data file originally in the first directory to another newly created directory in the local file system (called the second directory, which can be named after the current date). After the file rotation is complete, only the newly created data files remain in the first directory. The data files originally in the first directory are the data files that existed in the first directory before the file rotation was performed.

[0092] For example, taking the case of performing a file rotation process once a day (at a time interval of 24 hours), the first module creates a new directory () every day, namely the second directory. Assuming that there is only one data file A in the first directory, when performing the file rotation process,

[0093] Furthermore, in the above embodiment, the above method also includes: in the process of transferring any data file originally in the first directory to the second directory, writing the burial point data carried by each received reporting request associated with any data file into the any data file until the transfer process is completed.

[0094] In this embodiment, when performing file rotation, the handle of each data file in the first directory is saved in advance. When transferring any original data file in the first directory, each process still uses the handle of the original data file to write embedded data to the file until the file is completely transferred to the second directory.

[0095] Specifically, the first module can be responsible for executing the above-mentioned file rotation processing through a shell script that is executed at a scheduled time. After the transfer of any data file is completed, a signal is sent to each process to instruct it to write the newly received related tracking data into the file with the same name corresponding to any data file in the first directory.

[0096] This embodiment can ensure that the "data storage" process will not be interrupted during the file rotation process, thereby ensuring efficient reception of embedded data in high-concurrency and high-traffic scenarios.

[0097] It should be noted that, with respect to the various steps included in the data acquisition method provided in any of the above embodiments, unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and these steps may be executed in other orders. Furthermore, at least a portion of these steps may include multiple sub-steps or multiple stages, and these sub-steps or stages do not necessarily need to be completed at the same time, but may be executed at different times. The order of execution of these sub-steps or stages is not necessarily sequential, but may be executed in rotation or alternation with other steps or at least a portion of the sub-steps or stages of other steps.

[0098] Based on the same inventive concept, the present application also provides a service node. In some embodiments, Figure 6 As shown, the service node includes a first module 10 and a second module 20;

[0099] The first module 10 is configured to receive a report request forwarded by the load balancing server, store the tracking data carried in the report request in the local file system, and return a response message indicating a successful report to the user terminal that sent the report request;

[0100] The second module 20 is used to asynchronously send the tracking data stored in the local file system to the message queue.

[0101] In some embodiments, the first module 10 receives a reporting request forwarded by the load balancing server through the first module, and when storing the burial point data carried by the reporting request into the local file system, it is used to: create multiple processes and multiple disk drop task queues corresponding to the above-mentioned multiple processes; generate a corresponding disk drop task according to each reporting request forwarded by the load balancing server, and add the generated disk drop task to one of the above-mentioned multiple disk drop task queues; obtain the disk drop task from the corresponding disk drop task queue through each process, and write the burial point data carried by the reporting request indicated by the disk drop task into the local file system.

[0102] In some embodiments, there are multiple user terminals; the load balancing server provides multiple reporting interfaces for each user terminal, and each user terminal sends a reporting request carrying burial point data to the load balancing server through a target reporting interface among the above multiple reporting interfaces. The target reporting interface refers to a reporting interface corresponding to the business type of the burial point data carried by the reporting request; the first directory of the local file system is provided with multiple data files.

[0103] In some embodiments, when the first module 10 writes the burial point data carried by the target reporting request into the local file system through the target process, it is used to: obtain the interface identifier of the reporting interface used to send the target reporting request through the target process; wherein, the target process refers to any process, and the target reporting request refers to any reporting request; determine the target write data file corresponding to the interface identifier in the above-mentioned multiple data files according to the preset first correspondence relationship, and write the burial point data carried by the target reporting request into the target write data file; the preset first correspondence relationship includes the correspondence between the above-mentioned multiple reporting interfaces and the above-mentioned multiple data files.

[0104] In some embodiments, when the first module 10 writes the burial point data carried by the target reporting request into the target write data file through the target process, it is used to: query through the target process whether there is currently a handle to the target write data file; if so, write the burial point data carried by the target reporting request into the target write data file according to the handle of the target write data file; if not, request the local file system to open the target write data file to obtain the handle of the target write data file, save the obtained handle, and write the burial point data carried by the target reporting request into the target write data file according to the obtained handle.

[0105] In some embodiments, each data file in the first directory is provided with a corresponding file lock; before the first module 10 writes the buried point data carried by the target reporting request into the target write data file according to the handle of the target write data file, it is also used to: compete with other processes for the target file lock through the target process, and the target file lock refers to the file lock corresponding to the target write data file; if the target file lock is obtained, the next step is executed; if the target file lock cannot be obtained, the next step is waited.

[0106] In some embodiments, when the first module 10 asynchronously sends the burial point data stored in the local file system to the message queue through the second module, it is used to: create multiple threads through the second module; and send the burial point data in the above-mentioned multiple data files to the message queue through the above-mentioned multiple threads.

[0107] In some embodiments, the message queue is provided with multiple topics; when the first module 10 sends the embedded data in the target data file to the message queue through the target thread, it is used to: obtain the file identifier of the target data file through the target thread; the target thread refers to any thread, and the target data file refers to any data file in the first directory; determine the target topic corresponding to the file identifier among the above-mentioned multiple topics according to the preset second correspondence relationship, and send the embedded data in the target data file to the target topic; the preset second correspondence relationship includes the correspondence between the above-mentioned multiple data files and the above-mentioned multiple topics.

[0108] In some embodiments, the first module 10 is further configured to periodically create a file with the same name as each original data file in the first directory as a corresponding replacement file, and transfer each original data file to the second directory in the local file system.

[0109] In some embodiments, the first module 10 is also used to write the burial point data carried by each received reporting request associated with any data file into the any data file during the process of transferring the original data file in the first directory to the second directory until the transfer process is completed.

[0110] The specific definition of the service node can be found in the definition of the data collection method above and will not be repeated here. Each module in the above-mentioned service node can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0111] The present application also provides a data acquisition system, including a load balancing server, a server cluster consisting of multiple service nodes, and a message queue; each service node is provided with a first module and a second module; wherein:

[0112] The load balancing server is used to receive the report request sent by the user terminal and forward the report request to the service node in the server cluster;

[0113] Each service node is used to receive the reporting request forwarded by the load balancing server through the first module, store the tracking data carried in the reporting request in the local file system, return a response message indicating that the report is successful to the user terminal that sent the reporting request, and asynchronously send the tracking data stored in the local file system to the message queue through the second module.

[0114] The specific definition of the data acquisition system can be found in the definition of the data acquisition method above and will not be repeated here. Each module in the above-mentioned data acquisition system can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0115] In some embodiments, a computer device is provided, whose internal structure diagram can be as follows: Figure 7 shown.

[0116] The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as burial point data. The specific stored data can also be found in the definitions in the above method embodiments. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data acquisition method is implemented.

[0117] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0118] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data acquisition method provided in any of the above embodiments can be implemented.

[0119] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data acquisition method provided in any of the above embodiments are implemented.

[0120] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0121] Those skilled in the art will appreciate that all or part of the processes in the above method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data collection method, characterized in that: Applied to any service node in a server cluster, the service node includes a first module and a second module; the method includes: The first module receives the report request forwarded by the load balancing server, stores the tracking data carried in the report request in the local file system, and returns a response message indicating a successful report to the user terminal that sent the report request; Asynchronously sending the tracking data stored in the local file system to the message queue through the second module; Receiving, through the first module, a report request forwarded by the load balancing server, and storing the tracking data carried in the report request in a local file system, including: Creating multiple processes and multiple disk flushing task queues corresponding to the multiple processes; Generate a corresponding disk placement task according to each report request forwarded by the load balancing server, and add the generated disk placement task to one of the multiple disk placement task queues; Obtaining, by each of the processes, a disk placement task from a corresponding disk placement task queue, and writing the tracking data carried in the reporting request indicated by the disk placement task into the local file system; There are multiple user terminals; the load balancing server provides multiple reporting interfaces for each of the user terminals, and each of the user terminals sends a reporting request carrying burial point data to the load balancing server through a target reporting interface among the multiple reporting interfaces, and the target reporting interface refers to a reporting interface corresponding to the business type of the burial point data carried by the reporting request; The first directory of the local file system is provided with a plurality of data files; When the tracking data carried by the target reporting request is written into the local file system through the target process, it includes: Obtaining, through the target process, an interface identifier of a reporting interface for sending the target reporting request; wherein the target process refers to any of the processes, and the target reporting request refers to a reporting request forwarded by any of the load balancing servers; Determine the target write data file corresponding to the interface identifier in the multiple data files according to a preset first correspondence, and write the tracking data carried by the target reporting request into the target write data file; the preset first correspondence includes the correspondence between the multiple reporting interfaces and the multiple data files; Asynchronously sending the buried data stored in the local file system to the message queue through the second module, including: creating multiple threads through the second module; sending the buried data in the multiple data files to the message queue through the multiple threads; The message queue is provided with multiple topics; when the embedded data in the target data file is sent to the message queue through the target thread, it includes: obtaining the file identifier of the target data file through the target thread; the target thread refers to any of the threads, and the target data file refers to any of the data files in the first directory; determining the target topic corresponding to the file identifier among the multiple topics according to a preset second correspondence, and sending the embedded data in the target data file to the target topic; the preset second correspondence includes the correspondence between the multiple data files and the multiple topics.

2. The method according to claim 1, wherein When the target process writes the tracking data carried by the target reporting request into the target write data file, it includes: Querying, through the target process, whether there is currently a handle to the target write data file; If so, writing the buried point data carried by the target reporting request into the target writing data file according to the handle of the target writing data file; If not, request the local file system to open the target write data file to obtain the handle of the target write data file, save the obtained handle, and write the burial point data carried by the target reporting request into the target write data file according to the obtained handle.

3. The method according to claim 2, wherein Each of the data files in the first directory is provided with a corresponding file lock; Before writing the tracking data carried by the target reporting request into the target writing data file according to the handle of the target writing data file, the method further includes: The target process competes with other processes for a target file lock, where the target file lock refers to a file lock corresponding to the target write data file; If the target file lock is obtained, execute the next step; If the target file lock cannot be obtained, wait.

4. The method according to claim 1, wherein The method further comprises: The first module periodically creates a file with the same name as a corresponding replacement file for each original data file in the first directory, and transfers each original data file to the second directory in the local file system.

5. The method according to claim 4, wherein The method further comprises: During the process of transferring any data file originally in the first directory to the second directory, the burial point data carried by each received reporting request associated with the any data file will be written into the any data file until the transfer process is completed.

6. A data acquisition system, characterized in that: It includes a load balancing server, a server cluster consisting of multiple service nodes, and a message queue; each of the service nodes is provided with a first module and a second module; The load balancing server is configured to receive a report request sent by a user terminal and forward the report request to a service node in the server cluster; Each of the service nodes is configured to receive a report request forwarded by the load balancing server through a first module, store the tracking data carried in the report request in a local file system, return a response message indicating a successful report to the user terminal that sent the report request, and asynchronously send the tracking data stored in the local file system to the message queue through a second module; The first module receives the report request forwarded by the load balancing server, and stores the tracking data carried by the report request into the local file system, including: creating multiple processes and multiple disk drop task queues corresponding to the multiple processes; generating a corresponding disk drop task according to each report request forwarded by the load balancing server, and adding the generated disk drop task to one of the multiple disk drop task queues; obtaining the disk drop task from the corresponding disk drop task queue through each process, and writing the tracking data carried by the report request indicated by the disk drop task into the local file system; There are multiple user terminals; the load balancing server provides multiple reporting interfaces for each of the user terminals, and each of the user terminals sends a reporting request carrying burial point data to the load balancing server through a target reporting interface among the multiple reporting interfaces, and the target reporting interface refers to a reporting interface corresponding to the business type of the burial point data carried by the reporting request; The first directory of the local file system is provided with a plurality of data files; When the tracking data carried by the target reporting request is written into the local file system through the target process, it includes: obtaining the interface identifier of the reporting interface used to send the target reporting request through the target process; wherein, the target process refers to any of the processes, and the target reporting request refers to the reporting request forwarded by any of the load balancing servers; determining the target write data file corresponding to the interface identifier among the multiple data files according to a preset first correspondence, and writing the tracking data carried by the target reporting request into the target write data file; the preset first correspondence includes the correspondence between the multiple reporting interfaces and the multiple data files; Asynchronously sending the buried data stored in the local file system to the message queue through the second module, including: creating multiple threads through the second module; sending the buried data in the multiple data files to the message queue through the multiple threads; The message queue is provided with multiple topics; when the embedded data in the target data file is sent to the message queue through the target thread, it includes: obtaining the file identifier of the target data file through the target thread; the target thread refers to any of the threads, and the target data file refers to any of the data files in the first directory; determining the target topic corresponding to the file identifier among the multiple topics according to a preset second correspondence, and sending the embedded data in the target data file to the target topic; the preset second correspondence includes the correspondence between the multiple data files and the multiple topics.

Images