Data acquisition method and device, electronic equipment, storage medium and program product
By employing an asynchronous communication mechanism in the RDMA technology environment, data is written into a queue and managed by timestamps, solving the problems of low efficiency and insufficient real-time performance in traditional data acquisition schemes, and achieving efficient and stable data acquisition.
Patent Information
- Application Number
- CN202511093450.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-05
- Publication Date
- 2025-11-21
AI Technical Summary
Traditional data acquisition solutions are inefficient, lack real-time performance, and are prone to command execution failures that lead to data omissions. In particular, in complex business scenarios supported by RDMA technology, the problem of repeated data collection by multiple monitoring platforms seriously affects overall performance.
An asynchronous communication mechanism is adopted to write indicator data into the target queue according to the collection frequency and type. The data collection interval is managed by traversing the timestamps in the queue, so as to realize the asynchronous production and consumption of data, reduce latency, and improve reading efficiency and stability.
Effectively manage the data production and consumption process, reduce latency, improve the efficiency and stability of indicator data reading, ensure the real-time and completeness of data collection, and avoid duplicate collection and data omission.
Smart Images
Figure CN120994600A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, specifically to data acquisition methods, devices, electronic devices, storage media, and program products. Background Technology
[0002] With the increasing maturity of Remote Direct Memory Access (RDMA) technology and its large-scale deployment in data centers, it has become a high-performance communication solution for various high-performance business scenarios (such as storage, recommendation search, large model training, and large model inference).
[0003] Due to the high throughput and low latency performance characteristics of RDMA technology, the business scenarios supported by RDMA are becoming increasingly numerous and complex. Therefore, there is a need for more sensitive perception of network transmission status and timely task scheduling of nodes within the data center. This places higher demands on the real-time performance and collection cycle of monitoring data. Traditional data acquisition solutions use a blocking approach, triggering the collection of indicator data only when a command is issued. This results in low data acquisition efficiency and compromised real-time performance. Furthermore, various factors can cause command execution failures, leading to missed data collection. Summary of the Invention
[0004] In view of this, the present disclosure provides a data acquisition method, apparatus, electronic device, storage medium, and program product to solve the problem of low data acquisition efficiency.
[0005] In a first aspect, this disclosure provides a data acquisition method, including: acquiring data acquisition frequency and data acquisition type; writing the data of each indicator corresponding to the data acquisition type into a target queue according to the data acquisition frequency; acquiring the data acquisition interval; traversing the timestamps of each indicator data in the target queue, and reading the indicator data whose timestamps are located within the data acquisition interval from the target queue.
[0006] Secondly, this disclosure provides a data acquisition device, comprising: a first acquisition module for acquiring data acquisition frequency and data acquisition type; a writing module for writing various indicator data corresponding to the data acquisition type into a target queue according to the data acquisition frequency; a second acquisition module for acquiring data acquisition interval; and a traversal module for traversing the timestamps of various indicator data in the target queue and reading indicator data whose timestamps are located within the data acquisition interval from the target queue.
[0007] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the data acquisition method described in the first aspect or any corresponding embodiment.
[0008] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data acquisition method described in the first aspect or any corresponding embodiment thereof.
[0009] Fifthly, this disclosure provides a computer program product, including computer instructions for causing a computer to execute the data acquisition method described in the first aspect or any corresponding embodiment thereof.
[0010] The data acquisition method provided in this disclosure writes various indicator data corresponding to the data acquisition type into a target queue according to the data acquisition frequency, and then iterates through each indicator data in the target queue to read indicator data whose timestamps are located within the data acquisition interval. Therefore, this solution first temporarily stores the acquired indicator data in the target queue, and then reads the indicator data whose timestamps are located within the data acquisition interval from the target queue. That is, using an asynchronous communication mechanism, it can effectively manage the data production and consumption process, reduce data acquisition latency, improve the efficiency of indicator data reading, and ensure the stability and real-time performance of the indicator data reading process. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is an architecture diagram for traditional data acquisition methods;
[0013] Figure 2 Architecture diagram of the data acquisition method provided in the embodiments of this disclosure;
[0014] Figure 3 This is a flowchart illustrating a data acquisition method provided according to an embodiment of the present disclosure;
[0015] Figure 4 This is a flowchart illustrating another data acquisition method provided according to an embodiment of the present disclosure;
[0016] Figure 5This is a schematic diagram of reading and writing a circular queue according to an embodiment of this disclosure;
[0017] Figure 6 This is a schematic diagram of a data block provided according to an embodiment of this disclosure;
[0018] Figure 7 This is a schematic diagram illustrating the overwriting of data in a circular queue when it is full, according to an embodiment of this disclosure.
[0019] Figure 8 This is a flowchart illustrating another data acquisition method provided according to an embodiment of the present disclosure;
[0020] Figure 9 This is a schematic diagram illustrating the traversal of a circular queue according to an embodiment of this disclosure;
[0021] Figure 10 This is a schematic diagram illustrating the traversal of a circular queue by multiple consumers according to an embodiment of this disclosure;
[0022] Figure 11 This is an architecture diagram of a data acquisition method for large-scale model application scenarios provided in the embodiments of this disclosure;
[0023] Figure 12 This is a flowchart illustrating how a resident monitoring process reads indicator data from a hardware register or driver, according to embodiments of this disclosure.
[0024] Figure 13 This is a flowchart illustrating how a data acquisition tool, according to an embodiment of this disclosure, reads indicator data whose timestamps are located within a data acquisition interval from a target queue.
[0025] Figure 14 This is a schematic diagram of the structure of a data acquisition device provided according to an embodiment of the present disclosure;
[0026] Figure 15 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0033] With the increasing maturity and large-scale deployment of Remote Direct Memory Access (RDMA) technology in data centers, it has become a high-performance communication solution for various high-performance business scenarios (such as storage, recommendation search, large model training, and large model inference). As RDMA supports more and more complex business scenarios, it also brings more challenges and iterative evolution to its various technical aspects. Due to the high throughput and low latency characteristics of RDMA technology, a more sensitive perception of network transmission status is needed to enable timely task scheduling for nodes within the data center. Therefore, this places higher demands on the real-time performance and acquisition cycle of monitoring data.
[0034] like Figure 1The traditional data acquisition scheme shown involves a monitoring agent calling a custom tool (such as xxTool) to read metric data from storage devices, such as RDMA Network Interface Cards (RNICs), or a standardized tool (such as ethTool or ibv_devices) to read metric data from RDMA drives. However, this method uses a blocking approach, where the monitoring platform calls the custom or standardized tool only when a command is issued, resulting in low efficiency, unreliable real-time performance, and potential command failures leading to missed data collection. Furthermore, if multiple monitoring platforms need to use the same data, duplicate collection occurs, potentially impacting overall performance.
[0035] In view of this, the technical solution disclosed herein writes the indicator data corresponding to each data collection type into a target queue according to the data collection frequency, and then iterates through each indicator data in the target queue to read the indicator data whose timestamps are located within the data collection interval. Thus, this solution first temporarily stores the collected indicator data in the target queue, and then reads the indicator data whose timestamps are located within the data collection interval from the target queue. That is, using an asynchronous communication mechanism, it can effectively manage the data production and consumption process, reduce data collection latency, improve the efficiency of indicator data reading, and ensure the stability and real-time performance of the indicator data reading process.
[0036] As one optional application scenario of this disclosure embodiment, such as Figure 2 As shown, the monitoring agent can be deployed in corresponding electronic devices. The monitoring platform communicates with the monitoring agent to obtain the indicator data sent by the monitoring agent. The monitoring agent includes a data collection tool, a circular buffer, and a resident monitoring process. Specifically, the resident monitoring process can collect indicator data from hardware registers such as RNICs or drivers such as RDMA drivers according to the data collection frequency and data collection type, and write it to the circular buffer. In this case, the resident monitoring process can act as a data producer to collect data from the RNIC or RDMA. The data collection tool shares the circular buffer with the resident monitoring process. It can read indicator data with timestamps located in the data collection interval from the circular buffer according to the data collection interval. In this case, the data collection tool can act as a data consumer to consume the indicator data in the circular buffer. Finally, the data collection tool sends the collected indicator data to the monitoring agent, and the monitoring agent sends it to the monitoring platform. Both the monitoring agent and the data collection tool can be processes or services deployed in electronic devices. A queue can be set up in a circular buffer, and correspondingly, the indicator data collected by the resident monitoring process can be written to the queue in the circular buffer.
[0037] It should be understood that electronic devices can store and maintain data. Examples of electronic devices may include supercomputers, personal computers, laptops, in-vehicle computing devices, mobile devices (such as smartphones, tablets, etc.), or combinations thereof. It should be understood that the electronic devices described herein are merely exemplary and not limiting; other different types of electronic devices may also be used.
[0038] According to an embodiment of this disclosure, a data acquisition method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0039] This embodiment provides a data acquisition method that can be used in the aforementioned electronic device. Figure 3 This is a flowchart of a data acquisition method according to an embodiment of the present disclosure, such as... Figure 3 As shown, the process includes the following steps:
[0040] Step S301: Obtain the data acquisition frequency and data acquisition type.
[0041] Data collection frequency can be defined as the time interval between collecting indicator data. For example... Figure 2 The resident monitoring process shown can read indicator data from hardware registers such as RNIC or drivers such as RDMA drivers according to the data acquisition frequency.
[0042] Data collection types can be categorized by the type or attribute of the metrics to be collected. For example, in server monitoring scenarios, data collection types include hardware metrics, software metrics, and network metrics. Hardware metrics can include metrics such as Central Processing Unit (CPU) utilization, memory usage, disk I / O (input / output), and network card throughput. Software metrics can include the number of processes, service response time, and number of error logs. Network metrics can include the number of Transmission Control Protocol (TCP) connections, packet loss rate, and latency. In high-speed network scenarios, data collection types can be anomaly-related or performance-related. For instance, anomaly-related metrics can include TCP retransmission rate, packet loss rate, traffic bursts / drops, and the number of abnormal connections. Performance-related metrics can include total bandwidth utilization, network latency, one-way latency, port throughput, and round-trip latency.
[0043] The data acquisition frequency and type can be pre-set. For example, they can be pre-configured in a corresponding circular buffer. The resident monitoring process can then acquire data from hardware registers or drives and write it to the target queue based on the chosen frequency and type. Alternatively, if the monitoring agent has a user interface, the user can set the data acquisition frequency and type there. The monitoring agent can then generate data acquisition commands based on these commands. Upon receiving a data acquisition command, the resident monitoring process can acquire data from hardware registers or drives and write it to the target queue according to the specified frequency and type.
[0044] Step S302: Write the data of each indicator corresponding to the data collection type into the target queue according to the data collection frequency.
[0045] As a specific example, multiple circular buffers can be set up according to the data acquisition frequency and data acquisition type based on the needs of the monitoring platform. Each circular buffer can have a queue, so that a set of data acquisition frequencies and data acquisition types can correspond to a queue. The queue can be a lock-free queue such as a circular queue.
[0046] like Figure 2 The resident monitoring process shown can read various indicator data corresponding to the data acquisition type from hardware registers or drivers according to the data acquisition frequency, and then write each indicator data into the target queue corresponding to the data acquisition type. During the writing process, the resident monitoring process can first obtain the tail pointer of the target queue, and then write multiple indicator data into the target queue at once according to the direction the tail pointer points to the tail, or it can write the indicator data into the target queue one by one. After the writing is complete, the tail pointer is updated. It should be understood that since the resident monitoring process can act as a data producer in this scenario, the tail pointer of the target queue can also act as a producer pointer.
[0047] Step S303: Obtain the data collection range.
[0048] A data collection interval can be defined as the specific range and boundaries of various indicator data to be collected during the data collection process. The data collection interval can define the indicator data from multiple dimensions. For example, in the time dimension, the data collection interval can be a period of time in the past, such as reading indicator data from 10 seconds ago; in the numerical dimension, the data collection interval can be from second X to second Y, such as reading indicator data from second 1 to second 10.
[0049] Here, the monitoring agent can send data collection ranges to the data collection tool via commands. Alternatively, in scenarios where the monitoring agent has a user interface, the user can set the data collection range on the user interface, and then the monitoring agent will send the data collection range to the data collection tool.
[0050] Step S304: Iterate through the timestamps of each indicator data in the target queue and read the indicator data whose timestamps are within the data collection interval from the target queue.
[0051] The timestamp of the metric data can be the time when the resident monitoring process reads the metric data from the hardware register or driver; or the time when the resident monitoring process writes the read metric data to the target queue. It should be understood that the time information in the data acquisition interval and the timestamp of the metric data belong to the same time dimension.
[0052] This can be the timestamp of each indicator data in the target queue read one by one. If the timestamp of a certain indicator data is within the data collection period, then that indicator data is read.
[0053] The data acquisition method provided in this embodiment writes the indicator data corresponding to each data acquisition type into a target queue according to the data acquisition frequency, and then iterates through each indicator data in the target queue to read the indicator data whose timestamps are located within the data acquisition interval. Therefore, this scheme first temporarily stores the collected indicator data in the target queue, and then reads the indicator data whose timestamps are located within the data acquisition interval from the target queue. That is, using an asynchronous communication mechanism, it can effectively manage the data production and consumption process, reduce data acquisition latency, improve the efficiency of indicator data reading, and ensure the stability and real-time performance of the indicator data reading process.
[0054] This embodiment provides a data acquisition method that can be used in the aforementioned electronic device. Figure 4 This is a flowchart of a data acquisition method according to an embodiment of the present disclosure, such as... Figure 4 As shown, the process includes the following steps:
[0055] Step S401: Obtain the data acquisition frequency and data acquisition type. For details, please refer to [link to relevant documentation]. Figure 3 Step S301 of the illustrated embodiment will not be described again here.
[0056] Step S402: Write the data of each indicator corresponding to the data collection type into the target queue according to the data collection frequency.
[0057] Specifically, step S402 includes:
[0058] Step S4021: Encapsulate each indicator data according to a preset encapsulation method to obtain at least one data block.
[0059] The preset encapsulation method can be a pre-defined encapsulation method. For example, it can encapsulate various indicator data according to the same time window, the same monitored object, or the same type of indicator data. As a specific example, if four indicator data are collected in the same time window, and each indicator data occupies 10 bytes, then the total number of bytes occupied by the four indicator data is 40. Since one data block is set to 32 bytes, the four indicator data can be placed into two data blocks.
[0060] Step S4022: Obtain the current data tail of the target queue, and write at least one data block into the target queue in the direction of the data tail pointing to the tail.
[0061] The current tail of the target queue can be the position of the last data block in the current target queue. That is, starting from the tail, at least one data block is written into the target queue in the direction of the tail. As a concrete example, such as... Figure 5 As shown, the target queue is set as a circular queue, and pi is set as the producer pointer in the circular queue. As mentioned earlier, the resident monitoring process can first obtain the position of the producer pointer pi, and starting from the position of the producer pointer pi, write at least one data block to an empty position in the circular queue in the direction of the tail of the circular queue. After writing at least one data block to an empty position in the circular queue, the position of the producer pointer pi can be updated to the tail of the current circular queue, so that at least one data block can be written to the circular queue again based on the position of the producer pointer pi.
[0062] Encapsulating each indicator data according to a preset encapsulation method to obtain at least one data block ensures the orderliness and integrity of the data, reducing the risk of data corruption or loss. Then, writing at least one data block into the target queue in a tail-to-tail direction reduces the complexity of writing at least one data block to the target queue, while improving the compatibility, scalability, and processing efficiency of the data acquisition system.
[0063] In some optional implementations, step S4021 above includes:
[0064] Step a1: Obtain the descriptor for each indicator data. The descriptor includes one or more of the following: queue type, number of bytes occupied by a data block, total number of indicator data, number of bytes occupied by each indicator data, and timestamp.
[0065] Step a2 involves encapsulating each indicator data and descriptor to obtain at least one data block.
[0066] Descriptors can be metadata for various indicator data. They are used to define, interpret, or constrain the key attributes of the data blocks that are subsequently encapsulated, ensuring that the data blocks can be correctly identified, parsed, and processed. The specific descriptors are shown in Table 1.
[0067] Table 1 Descriptors
[0068]
[0069] In the process of encapsulating various indicator data and descriptors to obtain at least one data block, we can first obtain the number of bytes occupied by the descriptors of all indicator data collected in one go. Then, we combine the number of bytes occupied by the descriptors with the total number of bytes occupied by all indicator data collected in one go to determine the number of data blocks. Finally, we write the descriptors and all indicator data collected in one go into the corresponding data blocks sequentially. Following the previous example, if four indicator data points are collected in the same time window, and each indicator data point occupies 10 bytes, the total number of bytes occupied by the four indicator data points is 40. Since the number of bytes occupied by the descriptors is 10, the total number of bytes occupied is 50. If we set one data block to 32 bytes, then the four indicator data points can be placed into two data blocks.
[0070] As a specific example, such as Figure 6 As shown, the corresponding number of registers can be obtained according to the information in the descriptor, and at least one data block can be stored in the register, and then at least one data block can be stored in the circular queue.
[0071] Each indicator data and its corresponding descriptor are encapsulated to obtain at least one data block, which can support flexible expansion and compatibility with different data collection types, providing a foundation for subsequent data transmission and analysis.
[0072] In some optional implementations, the method further includes: obtaining the current head of the target queue; if the target queue is full, overwriting the existing data in the target queue with at least one data block in the direction from the head of the data queue to the tail of the queue, and writing at least one data block into the target queue.
[0073] In the standard producer-consumer model, when a consumer fails, it cannot consume data from the queue, causing the queue to remain full. This prevents producers from continuing to produce data, thus impacting the overall data acquisition process. Specifically, because consumers cannot process data in the queue in a timely manner, the queue quickly fills up. Once the queue is full, producers are blocked and cannot add new data. This not only affects the production of current data but may also lead to the loss of subsequent data or delays in acquisition, severely impacting the stability and reliability of the entire data acquisition system. In multi-consumer scenarios, since multiple consumers share a queue, if one consumer fails, causing the producer to be blocked, the entire data acquisition system will be in a blocked state. To address the problem of a full queue and support the access needs of multiple consumers, this disclosure provides a resident monitoring process and acquisition tools, hiding internal implementation details. It eliminates the concept of consumer pointers, maintaining only producer pointers. The resident monitoring process, acting as the data producer, only needs to focus on the producer pointer during data production, and can directly overwrite when the target queue is full. As a concrete example, such as... Figure 7 As shown, when the target queue is full, the producer pointer pi points to both the head and tail of the queue. From the direction of the producer pointer pointing to the tail of the queue, at least one data block can be used to overwrite the existing data in the target queue, thereby writing at least one data block into the target queue.
[0074] In scenarios where there are no free slots in the target queue, the data is directed from the head of the queue to the tail, and at least one data block is used to overwrite the existing data in the target queue. This ensures that the target queue always retains relatively new data, avoids blocking the producer's process, ensures that the data acquisition process is not interrupted, and reduces the problem of missing key information due to blocking.
[0075] Step S403: Obtain the data collection range. For details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.
[0076] Step S404: Iterate through the timestamps of each indicator data point in the target queue, and read the indicator data whose timestamps fall within the data collection period from the target queue. For details, please refer to [link to relevant documentation]. Figure 3 Step S304 of the illustrated embodiment will not be described again here.
[0077] The data acquisition method provided in this embodiment, if there is a free position in the target queue, writes at least one data block into the target queue in the direction from the tail of the data queue to the tail of the queue; if there is a free position in the target queue, uses at least one data block to overwrite the existing data in the target queue in the direction from the head of the data queue to the tail of the queue. This achieves the successful writing of at least one data block into the target queue, and this processing also makes the solution more robust and improves the success rate of data block writing.
[0078] This embodiment provides a data acquisition method that can be used in the aforementioned electronic device. Figure 8 This is a flowchart of a data acquisition method according to an embodiment of the present disclosure, such as... Figure 8 As shown, the process includes the following steps:
[0079] Step S801: Obtain the data acquisition frequency and data acquisition type.
[0080] Step S802: According to the data collection frequency, write the data of each indicator corresponding to the data collection type into the target queue. For details, please refer to [link to relevant documentation]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.
[0081] Step S803: Obtain the data acquisition range. For details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.
[0082] Step S804: Iterate through the timestamps of each indicator data in the target queue and read the indicator data whose timestamps are within the data collection interval from the target queue.
[0083] Specifically, step S804 includes:
[0084] Step S8041: Obtain the current tail of the target queue. See step S4022 for details.
[0085] Step S8042: With the tail of the data queue pointing to the head of the queue, read the indicator data whose timestamp is within the data collection interval from the target queue.
[0086] Starting from the tail of the data queue pointing towards the head, read the metric data whose timestamps fall within the data collection period from the target queue; that is, start reading in the opposite direction to where the metric data is written to the target queue. As a concrete example, such as... Figure 5As shown, we can start from the position of the producer pointer pi and traverse in reverse direction, i.e., with the producer pointer pointing to the head of the queue, to read the indicator data whose timestamps are within the data collection interval from the target queue. We can trace from the latest data to the older data, so as to quickly locate the indicator data that matches the data collection interval and avoid the inefficiency of skipping a large amount of irrelevant old data when traversing forward (from old to new).
[0087] In some optional implementations, step S8042 above includes:
[0088] Step b1: Obtain the current traversal depth corresponding to the current traversal position.
[0089] Step b2: If the current traversal depth is less than the target traversal depth, determine whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval.
[0090] Step b3: If the current position is located at the specified position, read the current indicator data corresponding to the current traversal position.
[0091] Step b4: Move to the next traversal position, update the next traversal position to the current traversal position, and proceed to the step of determining whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
[0092] The current traversal depth can be defined as the cumulative number of data blocks that have been traversed since the traversal started at the beginning of the traversal, such as the position of the producer pointer. The current traversal depth can be used to measure the progress of the current traversal operation and to determine whether to continue traversing.
[0093] The target traversal depth can be a pre-set traversal depth. Specifically, the target traversal depth can be half the total target queue depth; for example, if the total depth of the target queue is 10, then the target traversal depth can be 5. Of course, the target traversal depth is not limited to half the total target queue depth; it can be flexibly adjusted in conjunction with data acquisition frequency, data reading frequency, etc., and this disclosure does not impose any restrictions on it.
[0094] As a specific example, such as Figure 9 As shown, the monitoring agent calls the data collection tool to issue commands to collect the corresponding data collection interval's metric data by issuing commands in the form of data collection intervals. The data collection tool traverses the target queue from the position of the producer metric towards the head of the queue. When traversing to the position of elem1 (data block), it first determines whether the current traversal depth is less than the target traversal depth. If it is less and the timestamp of elem1 falls into the data collection interval (ta, tb), then elem1 is read. Then it moves to the next traversal position, such as the position of elem2, and continues to repeat the above process.
[0095] When the current traversal depth is less than the target traversal depth and the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval, the indicator data corresponding to the current traversal position can be read. This can avoid read-write conflicts caused by reading and writing to the same area of the target queue.
[0096] In some optional implementations, if the data is not located within the target traversal depth, determine whether the timestamp of the current indicator data is greater than or equal to the lower limit of the data collection interval; if the timestamp of the current indicator data is greater than or equal to the lower limit, move to the next traversal position and proceed to the step of determining whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
[0097] If the timestamp of the indicator data corresponding to the current traversal position is not within the data collection interval, the relationship between the current indicator data and the lower limit of the data collection interval is further determined. If the timestamp of the current indicator data is less than the lower limit, it indicates that the indicator data corresponding to the next traversal position is unlikely to be within the data collection interval, so the current process of reading indicator data can be terminated. If the timestamp of the current indicator data is greater than or equal to the lower limit, it indicates that the indicator data corresponding to the next traversal position may be within the data collection interval, so the process can be moved to the next traversal position and the above process can continue until there is a traversal position whose timestamp is less than the lower limit, at which point the current process of reading indicator data ends. As a specific example, if the data collection interval is (ta, tb), then the lower limit of the data collection interval can be ta.
[0098] In scenarios with multiple consumers, such as multiple monitoring agents, each monitoring agent can invoke the data collection tool and receive data collection ranges from a single data source, without needing to update the consumer pointer. For example... Figure 10 As shown, Monitoring Agent1, Monitoring Agent2, and Monitoring Agent3 can invoke the data collection tool and simultaneously read data blocks located in their respective data collection intervals from the target queue without updating the consumer pointer. The data collection interval corresponding to Monitoring Agent1 is (ta, tb), the data collection interval corresponding to Monitoring Agent2 is (tc, td), and the data collection interval corresponding to Monitoring Agent3 is (tx, ty).
[0099] In some optional implementations, the method further includes: after reading indicator data whose timestamps are within the data collection interval from the target queue, analyzing the read indicator data and the timestamps of each indicator data to generate indicator analysis results.
[0100] Here, after reading indicator data whose timestamps fall within the data collection period from the target queue, the read indicator data can be sent to the monitoring platform. The monitoring platform then analyzes the read indicator data based on the timestamps of each indicator, generating indicator analysis results. As a concrete example, the indicator analysis results can be displayed in the form of charts. Furthermore, if the monitoring platform has a user interface such as a monitoring dashboard, the indicator analysis results can also be sent to the monitoring dashboard for display.
[0101] The data acquisition method provided in this embodiment reads indicator data whose timestamps fall within the data acquisition interval from the target queue in the direction from the tail of the data queue to the head of the queue. This ensures that the read and write areas do not overlap, avoiding problems such as dirty reads and incomplete data caused by the consumer reading the same piece of data while the producer is writing. In addition, lock-free read and write are achieved through reverse traversal, which not only simplifies the logic but also improves concurrency efficiency.
[0102] As a specific application embodiment of this disclosure, such as Figure 11 As shown, in the application scenario of large model training, each server deploys model files and a monitoring agent. The monitoring agent collects network-related metrics data during the training process of the large model. The monitoring platform receives the metrics data from each monitoring agent, analyzes the data according to its timestamp, and displays the analysis results on the monitoring dashboard. Each monitoring agent includes a data collection tool, a circular buffer, and a resident monitoring process. For a single server, the specific process of data collection using the monitoring agent includes: the resident monitoring process reads various metrics data from hardware registers or drivers according to the data collection frequency and type, and acts as a producer to add data to the circular queue. The data collection tool shares the circular queue with the resident monitoring process and consumes the data from the queue. This asynchronous communication mechanism makes the data collection system transparent. When the monitoring agent calls the data collection tool to read metrics data from the circular queue, it does not need to know the specific storage structure of the metrics data, i.e., it does not need to be aware of the internal implementation logic, further simplifying the process of the monitoring agent sending metrics data.
[0103] like Figure 12 The diagram shows the resident monitoring process reading various indicator data from hardware registers or drivers and storing them in a circular queue, specifically including steps S1201 to S1207.
[0104] Step S1201: Create a circular queue, initialize the queue depth, and set the producer pointer to 0.
[0105] Step S1202: Set the data acquisition frequency and data acquisition type.
[0106] Step S1203: Collect multiple indicator data corresponding to the data acquisition type from hardware registers and drivers according to the data acquisition frequency.
[0107] Step S1204: Determine if an error has occurred. If no error has occurred, proceed to step S1205; if an error has occurred, proceed to step S1207. It should be understood that the error here could be a resident monitoring process or a server crashing, etc.
[0108] Step S1205: Construct descriptors for multiple indicator data collected at one time, encapsulate multiple indicator data and descriptors to obtain at least one data block and add it to a circular queue.
[0109] Step S1206: Update the producer pointer and repeat steps S1202 to S1206.
[0110] Step S1207: Report an alarm and exit.
[0111] like Figure 13 The diagram shows the process of a monitoring agent calling a data collection tool to read data blocks from a circular queue, specifically including steps S1301 to S1307.
[0112] Step S1301: The monitoring agent sets the data collection range and sends commands to the collection tool in the form of the data collection range by calling the collection tool.
[0113] Step S1302: The collection tool starts traversing from the producer pointer.
[0114] Step S1303: Obtain the current traversal depth corresponding to the current traversal position, and determine whether the current traversal depth is greater than or equal to the target traversal depth. If the current traversal depth is less than the target traversal depth, proceed to steps S1304 to S1307; if the current traversal depth is greater than or equal to the target traversal depth, end the process of reading data blocks from the circular queue.
[0115] Step S1304: Read the current data block corresponding to the current traversal position.
[0116] Step S1305: Check if the timestamp of the current data block corresponding to the current traversal position is within the data collection interval. If the timestamp of the current data block is within the data collection interval, the data block is sent to the monitoring Agent, i.e., step S1307 is executed, and steps S1303 to S1307 are continued; if the timestamp of the current data block is not within the data collection interval, step S1306 is executed.
[0117] Step S1306: Determine whether the timestamp of the current data block is greater than or equal to the lower limit of the data acquisition interval. If the timestamp of the current data block is greater than or equal to the lower limit of the data acquisition interval, the process of reading data blocks from the circular queue ends; if the timestamp of the current data block is less than the lower limit of the data acquisition interval, continue to execute steps S1303 to S1307.
[0118] Step S1307: Send the data block to the monitoring agent.
[0119] Through the above process, the circular queue structure can effectively manage the data production and consumption process and reduce data acquisition latency; using an asynchronous information communication mechanism, data producers and data consumers do not need to be explicitly synchronized, which greatly improves the system's concurrent processing capability; the same data can support multiple consumers to consume simultaneously, improving the scalability and flexibility of the data acquisition system.
[0120] This embodiment also provides a data acquisition device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0121] This embodiment provides a data acquisition device, such as... Figure 14 As shown, it includes:
[0122] The first acquisition module 1401 is used to acquire the data acquisition frequency and data acquisition type.
[0123] The writing module 1402 is used to write the data of each indicator corresponding to the data collection type into the target queue according to the data collection frequency.
[0124] The second acquisition module 1403 is used to acquire the data collection range.
[0125] Traversal module 1404 is used to traverse the timestamps of each indicator data in the target queue and read indicator data whose timestamps are within the data collection interval from the target queue.
[0126] In some alternative implementations, the write module 1402 includes:
[0127] The encapsulation unit is used to encapsulate each indicator data according to a preset encapsulation method to obtain at least one data block.
[0128] The write unit is used to obtain the current data tail of the target queue and write at least one data block into the target queue in the direction of the data tail pointing to the tail.
[0129] In some alternative implementations, the packaging unit includes:
[0130] The first acquisition subunit is used to acquire descriptors for each indicator data. The descriptor includes one or more of the queue type, the number of bytes occupied by a data block, the total number of indicator data, the number of bytes occupied by each indicator data, and the timestamp.
[0131] The encapsulation subunit is used to encapsulate various indicator data and descriptors to obtain at least one data block.
[0132] In some alternative embodiments, the device further includes:
[0133] The third acquisition module is used to acquire the current head of the target queue.
[0134] The overwrite module is used to overwrite the existing data in the target queue with at least one data block in the direction from the head of the data queue to the tail of the queue if the target queue is full.
[0135] In some alternative implementations, the traversal module 1404 includes:
[0136] The retrieval unit is used to retrieve the current tail of the target queue.
[0137] The reading unit is used to read indicator data whose timestamps are within the data collection interval from the target queue, with the data queue tail pointing to the queue head.
[0138] In some optional implementations, the reading unit includes:
[0139] The second sub-unit is used to obtain the current traversal depth corresponding to the current traversal position;
[0140] The sub-unit is determined to determine whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
[0141] Read sub-cells, which are used to read the current indicator data corresponding to the current traversal position if the current position is located at the specified location.
[0142] The moving sub-unit is used to move to the next traversal position, update the next traversal position to the current traversal position, and proceed to the step of determining whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
[0143] In some alternative embodiments, the device further includes:
[0144] The determination module is used to determine whether the timestamp of the current indicator data is greater than or equal to the lower limit of the data collection interval if the data is not located within the specified range.
[0145] The loop module is used to move to the next traversal position if the timestamp of the current indicator data is greater than or equal to the lower limit value, and then proceed to the step of determining whether the timestamp of the indicator data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
[0146] In some optional implementations, the device further includes an analysis module for analyzing the read indicator data and generating indicator analysis results based on the read indicator data and the timestamps of each indicator data after reading indicator data with timestamps located within the data collection interval from the target queue.
[0147] The data acquisition device provided in this disclosure can execute the data acquisition method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method. This solution first temporarily stores the collected indicator data in a target queue, and then reads the indicator data whose timestamps are located within the data acquisition interval from the target queue. That is, using an asynchronous communication mechanism can effectively manage the data production and consumption process, reduce data acquisition latency, improve the efficiency of indicator data reading, and ensure the stability and real-time performance of the indicator data reading process. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.
[0148] Figure 15 This is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. See below for details. Figure 15 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1502 or a program loaded from memory 1508 into random access memory (RAM) 1503. The RAM 1503 also stores various programs and data required for the operation of the electronic device. The processor 1501, ROM 1502, and RAM 1503 are interconnected via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.
[0149] Typically, the following devices can be connected to I / O interface 1505: input devices 1506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 1508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1509. Communication device 1509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0150] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1509, or installed from a memory 1508, or installed from a ROM 1502. When the computer program is executed by the processor 1501, it performs the functions defined above in the data acquisition method of embodiments of this disclosure.
[0151] Figure 15 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0152] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data acquisition method shown in the above embodiments is implemented.
[0153] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0154] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A data acquisition method, characterized by, The method comprises the following steps: acquiring a data collection frequency and a data collection type; writing each index data corresponding to the data collection type into a target queue according to the data collection frequency; acquiring a data collection interval; traversing a timestamp of each index data in the target queue, and reading the index data whose timestamp is within the data collection interval from the target queue.
2. The method of claim 1, wherein, The method of writing each index data corresponding to the data collection type into a target queue according to the data collection frequency comprises the following steps: packing each index data according to a preset packing mode to obtain at least one data block; acquiring a current data queue tail of the target queue, and writing the at least one data block into the target queue in a direction in which the data queue tail points to a queue tail.
3. The method of claim 2, wherein, The method of packing each index data according to a preset packing mode to obtain at least one data block comprises the following steps: acquiring a descriptor for each index data, wherein the descriptor comprises one or more of a queue type, an occupied byte number corresponding to one data block, a total number of the index data, and an occupied byte number corresponding to each index data and the timestamp; packing each index data and the descriptor to obtain the at least one data block.
4. The method of claim 2, wherein, The method further comprises the following steps: acquiring a current data queue head of the target queue; if the target queue is full, covering stored data in the target queue with the at least one data block in a direction in which the data queue head points to a queue tail, and writing the at least one data block into the target queue.
5. The method of claim 1, wherein, The method of traversing a timestamp of each index data in the target queue, and reading the index data whose timestamp is within the data collection interval from the target queue comprises the following steps: acquiring a current data queue tail of the target queue; reading the index data whose timestamp is within the data collection interval from the target queue in a direction in which the data queue tail points to a queue head.
6. The method of claim 5, wherein, The method of reading the index data whose timestamp is within the data collection interval from the target queue in a direction in which the data queue tail points to a queue head comprises the following steps: acquiring a current traversal depth corresponding to a current traversal position; if the current traversal depth is less than a target traversal depth, determining whether a timestamp of index data corresponding to the current traversal position is within the data collection interval; if yes, reading a current index data corresponding to the current traversal position; moving to a next traversal position, updating the next traversal position as the current traversal position, and entering the step of determining whether the timestamp of the index data corresponding to the current traversal position is within the data collection interval if the current traversal depth is less than the target traversal depth.
7. The method of claim 6, wherein, The method further comprises the following steps: if no, determining whether the timestamp of the current index data is greater than or equal to a lower limit value of the data collection interval; If the timestamp of the current index data is greater than or equal to the lower limit value, moving to the next traversal position and entering a step of determining whether the timestamp of the index data corresponding to the current traversal position is located in the data collection interval if the current traversal depth is less than a target traversal depth.
8. The method according to any one of claims 1 to 7, characterized in that, After reading the index data with the timestamp located in the data collection interval from the target queue, the method further comprises: According to the read index data and the timestamp of each index data, analyzing the read index data to generate an index analysis result.
9. A data acquisition device, characterized by Comprise: A first acquisition module is configured to acquire a data collection frequency and a data collection type; A writing module is configured to write each index data corresponding to the data collection type into a target queue according to the data collection frequency; A second acquisition module is configured to acquire a data collection interval; A traversal module is configured to traverse the timestamp of each index data in the target queue, and read the index data with the timestamp located in the data collection interval from the target queue.
10. An electronic device, comprising: Comprise: A memory and a processor are communicatively connected between each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data collection method in any one of claims 1 to 8.
11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the data collection method in any one of claims 1 to 8.
12. A computer program product, characterised in that, The computer instructions are used to make the computer execute the data collection method in any one of claims 1 to 8.