A high-efficiency data sampling and processing method and computer equipment

Abstract

The invention discloses an efficient data sampling and processing method and computer equipment, which configures a circular buffer queue and a flip buffer queue; creates parallel data reading and writing threads: reads instant sampling data from a hardware memory and writes it into the circular buffer queue in sequence; reads instant sampling data from the circular buffer queue in sequence according to the reading recorder information of each channel, updates the buffer data of the designated channel, and pushes the updated data to the forward buffer area in the flip buffer queue; data processing thread: cyclically reads data in the backward buffer area from the flip buffer queue, and after the reading is completed, the backward buffer area and the forward buffer area are flipped, and the read raw data is calculated and processed into the final data presented to the user. Not only does it solve the problem of inconsistent hardware data acquisition and software processing speed, but it also improves the overall data processing speed.

Description

Technical Field

[0001] The present invention relates to the technical field of instrument data acquisition and processing, and in particular to an efficient data sampling and processing method and computer equipment. Background Art

[0002] The data sampling process generally uses hardware to sample data and store it in memory. The software system then reads the data from the hardware storage and writes it to the software cache. The software then takes the data out of the software cache for processing. The entire process is generally a single-threaded processing process. Single-threaded processing has low data processing efficiency and cannot balance the speed difference between hardware sampling and software processing data.

[0003] In the prior art, in order to process data efficiently, the data processing efficiency is generally improved by creating multi-threaded parallel execution. The underlying hardware is responsible for collecting data, and the upper-level software is responsible for processing data. However, there is always a certain speed difference between the two. If the hardware collection speed is too fast for the software to handle, it may cause memory overflow or data confusion. Therefore, it is necessary to solve the problem of low efficiency of single-threaded data processing and the inconsistency between hardware collection and software processing speed. Summary of the invention

[0004] The purpose of the present invention is to provide an efficient data sampling and processing method and computer equipment to solve the problems of low efficiency of single-threaded data processing and inconsistency between hardware data sampling and software processing speed, thereby realizing high-speed data sampling and processing.

[0005] On the one hand, the present application provides an efficient data sampling and processing method, comprising the following steps:

[0006] Create data reading and writing threads and data processing threads that run in parallel, configure the circular buffer queue, flip buffer queue and reading recorders for each channel;

[0007] The processing of the data reading and writing thread includes: cyclically reading the instant sampling data from the hardware memory and writing it into the circular buffer queue in a sequential order, reading the instant sampling data from the circular buffer queue into the channel buffer data queue in sequence according to the reading recorder information of each channel, updating the buffer data in the specified channel buffer data queue (from the specified length at the specified position), and pushing the updated buffer data to the forward buffer area of ​​the flip buffer queue, waiting for processing by the data processing thread;

[0008] The processing process of the data processing thread includes: cyclically reading updated cache data from the backward cache area in the flip cache queue, flipping the forward cache area and the backward cache area, and calculating and processing the read data into final data presented to the user.

[0009] In the prior art, in order to process data efficiently, the data processing efficiency is generally improved by creating multi-threaded parallel execution; the underlying hardware is responsible for collecting data, and the upper-level software is responsible for processing data, but there is always a certain speed difference between the two. If the hardware collection speed is too fast for the software to handle, it may cause memory overflow or data confusion.

[0010] In view of this, the present application uses a circular cache queue and a flip cache queue to separate the software's reading and writing process from the processing process. The data reading and writing thread cyclically reads data from the hardware and writes it into the circular cache queue. Then, according to the reading recorder of each channel, it reads data from the circular cache queue according to a certain control logic, updates the data cache of the channel, and writes the updated channel cache data into the forward cache area of ​​the flip cache queue for the data processing thread to process the data. If the hardware acquisition speed is slow and no data is read, the channel cache data will be directly written into the flip cache queue; if the hardware acquires data at a fast speed and the software cannot handle it, when reading data from the circular cache queue, the most recent 3 frames of complete cache data will be saved, and other data will be erased as a whole frame. This not only ensures that the data processing thread always processes the latest data, but also balances the speed difference between software and hardware.

[0011] In an optional implementation, the process of configuring the circular cache queue includes setting the queue's head pointer, tail pointer, start pointer, and total buffer length, monitoring the position status of the head pointer and tail pointer in real time, calculating the cached length and the remaining writable length in real time, and before writing the instant sampling data, calculating the cached length, that is, the length of the currently unprocessed sampling data, based on the head pointer and tail pointer, and judging whether the cached length exceeds a preset threshold. If so, judging that there is redundant data in the circular cache queue, retaining the latest instant sampling data, and retaining the latest instant sampling data. The amount of data needs to meet the amount of data of the length of a channel cache data queue to ensure that a channel cache data queue can be filled. Then, the redundant data in the circular cache queue is erased, and the position status of the head pointer and tail pointer, the cached length, and the remaining writable length are updated to balance the speed difference between the data reading and writing threads and the data processing threads.

[0012] In an optional implementation manner, the specific process of writing the sampled data into the ring buffer queue includes:

[0013] According to the position of the tail pointer, the remaining writable length of the ring buffer queue is calculated. If the length of the real-time sampling data that can be read in the hardware memory is less than the remaining writable length, the real-time sampling data is written into the ring buffer queue, and the tail pointer position is refreshed to the position after writing;

[0014] If the length of the real-time sampling data that can be read in the hardware memory exceeds the remaining writable length, the real-time sampling data is divided into two parts and written, one part is written to the buffer area corresponding to the remaining writable length, and the other part of the data is written from the beginning of the buffer area corresponding to the start pointer, and the tail pointer position is refreshed to the position after writing;

[0015] After writing the instant sampling data, the position status of the head pointer and tail pointer of the circular buffer queue, the buffered length, and the remaining writable length are updated.

[0016] In an optional implementation manner, the preset threshold is three times the sum of the channel cache data queue lengths of all channels.

[0017] In some optional embodiments, each channel is preset with a unique reading recorder for controlling the update logic of the channel's cached data, and the reading recorder is used to record the queue length of the channel cache data queue, the length of written data, the remaining updateable data length and the current measurement position.

[0018] In an optional implementation, the process of reading data from the ring buffer queue, updating the channel buffer data queue, and writing it into the flip buffer queue is as follows:

[0019] Each channel is preset with a unique reading recorder to control the update logic of the channel cache data;

[0020] The data reading and writing thread will cyclically obtain the data currently collected by the hardware from the circular buffer queue. The length of each reading will not exceed the remaining updateable data length of the currently activated reading recorder. Combined with the current measurement position recorded in the reading recorder, the channel buffer data queue is updated in a fixed order to refresh the relevant information of the current reading recorder. If the data of the last point of the current channel is updated, it will switch to the next channel for reading. If it is already the last channel, it will switch to the first channel for reading. For each channel, the update process is:

[0021] Get the remaining updateable data length and current measurement position activated in the reading recorder of the current writing channel;

[0022] The instantaneous sampling data of the remaining updateable data length is read from the circular buffer queue, and the instantaneous sampling data is sequentially written into the channel buffer data queue of the current channel starting from the current measurement position, and the information recorded by the current reading recorder is refreshed.

[0023] After the cache data of the channel is updated, the latest cache data of the channel will be written to the forward buffer area of ​​the flip buffer queue, which will be used as the data source for subsequent data processing threads.

[0024] In an optional implementation, the channel cache data queue is a queue with a fixed length, and the queue length corresponds to the data length of the forward and backward cache areas in the flip cache queue.

[0025] In an optional implementation, when updating the designated channel cache data queue, the updateable data amount is calculated based on the current measurement position, the written data length and the remaining updateable data length of the matched reading recorder, combined with the total amount of data corresponding to the cached length in the circular cache queue;

[0026] If the cached length in the ring buffer queue is less than the remaining updateable data length, all data corresponding to the cached length in the ring buffer queue are stored in the specified channel buffer data queue;

[0027] If the cached length in the ring buffer queue is greater than the remaining updateable data length, only the amount of data corresponding to the remaining updateable data length is read and stored in the specified channel buffer data queue.

[0028] In an optional implementation, the flip cache queue includes a forward cache area and a backward cache area. The forward cache area is responsible for writing newly collected data by the data reading and writing thread, while the backward cache area is read and flipped by the data processing thread. The writing of the forward cache area and the reading of the backward cache area are controlled by the same thread lock, thereby ensuring that the thread timing does not conflict.

[0029] In a second aspect, the present application provides a computer device, comprising: at least one processor; and a memory, wherein the memory stores computer instructions that can be executed on the processor, and when the instructions are executed by the processor, an efficient data sampling and processing method of the first aspect is implemented.

[0030] The present invention has the beneficial effects:

[0031] The circular cache queue and flip cache queue are used to separate the software's reading and writing process from the processing process. The read-write thread cyclically reads data from the hardware and writes it to the circular cache queue. Then, according to the reading recorder, it reads data from the circular cache queue according to a certain control logic, updates the channel data cache, and writes the updated channel cache data to the forward cache area of ​​the flip cache queue for the data processing thread to process the data. If the hardware acquisition speed is slow and the read-write thread does not read any data, the channel cache data will be directly written to the flip cache queue; if the hardware data acquisition speed is fast and the read-write thread cannot handle it, when reading data from the circular cache queue, the latest stored several frames of complete channel cache data will be saved, and other data will be erased as a whole frame. This not only ensures that the data processing thread always processes the latest data, but also balances the speed difference between software and hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a flow chart of an efficient data sampling and processing method provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of data storage relationship provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a data read and write thread processing flow provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of a flow chart of a pointer moving with data in a ring buffer queue provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness.One of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0040] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] Example 1

[0043] like Figure 1 , Figure 2As shown, this embodiment provides an efficient data sampling processing method, comprising the following steps:

[0044] Create data reading and writing threads and data processing threads that run in parallel, configure the circular cache queue and flip cache queue;

[0045] The process of configuring the circular cache queue includes setting the queue's head pointer, tail pointer, start pointer, and total buffer length, monitoring the position status of the head pointer and tail pointer in real time, and calculating the cached length and the remaining writable length in real time. The flip cache queue includes a forward cache area and a backward cache area.

[0046] like Figure 3 As shown, the processing process of the data reading and writing thread includes:

[0047] S1, cyclically read the real-time sampling data from the hardware memory and write it into the ring buffer queue. The specific process includes:

[0048] S11, before writing the instant sampling data, the length of the currently unprocessed sampling data, that is, the cached length, is calculated according to the head pointer and the tail pointer, and whether there is redundant data is determined in combination with the queue length of the channel cache data queue (that is, the complete channel data length) and the preset threshold. If there is, the latest 3 frames of data corresponding to the complete channel data length will be retained, and other redundant data will be erased to balance the speed difference between the data reading and writing thread and the data processing thread; the preset threshold is three times the sum of the channel cache data queue lengths of all channels;

[0049] S12, according to the position of the tail pointer, calculate the remaining writable length of the circular buffer queue, if the length of the real-time sampling data that can be read in the hardware memory is less than the remaining writable length, write the real-time sampling data into the buffer queue, and refresh the tail pointer position to the corresponding position;

[0050] S13, if the length of the real-time sampling data that can be read in the hardware memory exceeds the remaining writable length, the real-time sampling data is divided into two parts and written, one part is written to the buffer area corresponding to the remaining writable length, and the other part of the data is written from the beginning of the buffer area corresponding to the start pointer, and the tail pointer position is refreshed to the position after writing; the pointer movement process is as follows Figure 4 As shown;

[0051] S14, after writing the instant sampling data, updating the position status of the head pointer and tail pointer of the circular buffer queue, the buffered length and the remaining writable length.

[0052] S2. After executing S1, data is read from the circular cache queue to update the channel cache data queue and write it to the flip cache queue. Each channel is preset with a channel cache data queue and a unique reading recorder. The reading recorder is used to control the update logic of the channel cache data queue. When reading data from the circular cache queue, the channel cache data queue of each channel is updated in sequence according to the index order of the channel. The reading recorder controls which channel pushes the data to the flip cache queue for reading. Only one channel will be read each time. The data reading and writing thread will cyclically obtain the data currently collected by the hardware from the circular cache queue. The length of each reading will not exceed the remaining updateable data length of the currently activated reading recorder. Combined with the current measurement position recorded in the reading recorder, the channel cache data queue is updated in a fixed order to refresh the relevant information of the current reading recorder. If the data of the last point of the current channel is updated, the remaining data in the circular cache queue will be switched to the next channel for reading. If it is already the last channel, it will switch to the first channel for reading.

[0053] After the cache data of each channel is updated, the latest channel cache data will be written into the forward cache area of ​​the flip cache queue to be used as the data source for subsequent data processing threads.

[0054] Repeat the above process, and in the above process, when reading or writing data, it is necessary to reset the state of the ring buffer queue and modify the head pointer and tail pointer. When writing data, the position pointed by the tail pointer moves with the position where the data is written, and when reading data, the position pointed by the head pointer moves with the position where the data is read. Figure 4 As shown, the state changes of the ring buffer queue during data writing and reading include the changes of the head pointer and the tail pointer. The current ring buffer queue is in the first state. At this time, the remaining writable length is not enough to write the data read from the hardware. Therefore, the data can be stored in the current remaining writable length first, and then the tail pointer is moved to the position of the start pointer, and then the remaining data is written from the position of the start pointer. After writing, the position of the tail pointer moves to the position corresponding to the length of the remaining data. At this time, the ring buffer queue is in the second state.

[0055] For each channel, the update process is:

[0056] S21, obtaining the remaining updateable data length, the written data length and the current measurement position activated in the reading recorder of the current writing channel, and calculating the updateable data amount in combination with the total amount of data corresponding to the cached length in the circular cache queue;

[0057] S22, if the cached length in the circular buffer queue is less than the remaining updateable data length, read the real-time sampling data of the remaining updateable data length from the circular buffer queue, write the real-time sampling data into the channel buffer data queue of the current channel in sequence starting from the current measurement position, and refresh the information recorded by the current reading recorder;

[0058] S23. If the cached length in the circular buffer queue is greater than the remaining updateable data length, only the amount of data corresponding to the remaining updateable data length is read, and the instant sampling data is written into the channel buffer data queue of the current channel in sequence starting from the current measurement position, and the information recorded by the current reading recorder is refreshed. After the data of the last point of the current channel is updated, the remaining data in the circular buffer queue will be switched to the next channel for updating. If it is already the last channel, it will be switched to the first channel for updating.

[0059] The processing of the data processing thread includes:

[0060] S3, the data processing thread will cyclically read the channel data in the backward buffer area from the flip buffer queue.

[0061] S4, after the reading is completed, the backward buffer area and the forward buffer area are flipped, and the read raw data is calculated and processed into the final data presented to the user. Since the channel cache data queue is a queue of fixed length (that is, the channel cache data queue length of all channels is the same), and corresponds to the data length of the forward and backward buffer areas in the flipped cache queue.

[0062] Since it involves multiple threads, the data reading and writing thread is responsible for obtaining the data collected by the hardware in real time, while the data processing thread is responsible for data processing. Therefore, there must be a speed difference between the two threads. The data reading and writing thread will first update the real-time sampling data read to the ring buffer queue. If the hardware acquisition speed is faster, there must be a lot of data in the ring buffer queue that cannot be processed in the future. At this time, it is necessary to erase some expired data, and then read the relatively new sampling data and update it to the channel buffer data queue, and push the data in the channel buffer data queue to the flip buffer queue. The data in the forward buffer area of ​​the flip buffer queue may not be processed in the future, so the update of the flip buffer queue will be skipped directly, and the next reading will be performed directly to ensure that the reading efficiency keeps up with the hardware acquisition speed. If the current software processing speed is faster, even if the data reading and writing thread does not read any data from the hardware, the data reading and writing thread will still push the cached data of the channel buffer data queue to the forward buffer area of ​​the flip buffer queue to ensure the normal operation of the data processing thread.

[0063] Exemplarily, assuming that the preset data volume of the channel cache data queue, that is, the queue length, is 200, then the size of each cache area of ​​the flip cache queue is also 200, and the data read and write threads will cyclically read the real-time data collected by the hardware from the FIFO, and then write the data into the circular cache queue. It is necessary to determine whether the amount of data currently to be written into the circular cache queue exceeds the remaining writable length of the circular cache queue. For example, the current amount of data collected is 601, and the remaining writable length of the circular cache queue is 100 at this time. Then, the amount of data of 601 needs to be split into two parts, and one part is written in sequence according to 100 to the queue position pointed to by the tail pointer corresponding to the remaining writable length, and then the remaining 401 data is written from the position pointed to by the start pointer. The data are written into the queue one by one, and the tail pointer is moved to the address after the data of 401 is written. At this time, the cached length in the cache queue is calculated (which may include historical cached data). If the cached length exceeds 600, it means that many complete measurement data frames are accumulated in the circular cache queue. Therefore, some obsolete data need to be erased, and at least 3 frames of the latest preset data volume (between 600 and 800, the data volume of one queue length) need to be retained. Expired data that is an integer multiple of the preset data volume need to be erased, and the head pointer is moved to the position after erasure. Then, data is read from the position of the head pointer and updated to the channel cache data queue. Then, the whole packet of data in the channel cache data queue is pushed to the forward buffer of the flip cache queue.

[0064] The process of the data processing thread executed in parallel when executing the data reading and writing thread is as follows: in order to improve the efficiency of the software in processing data, a flip cache queue is set, and the sizes of the forward cache area and the backward cache area of ​​the flip queue are equal, both of which are lengths corresponding to the preset data volume (200). When writing data to the flip cache queue, the status of the forward cache area and the backward cache area can be judged first. If there is still data in the forward cache area that has not been processed at this moment, the update of the flip cache queue will be skipped, and the next round of reading will be directly performed to overwrite the forward cache area with the latest data read from the channel cache queue; if the forward cache area is empty, new data will be written and the next round of reading will be started, while the data processing thread always only reads data from the backward buffer. After reading, the flip cache queue will be flipped, and then the read data will be processed.

[0065] Example 2

[0066] This embodiment provides a computer device, including: at least one processor; and a memory, wherein the memory stores computer instructions that can be run on the processor, and when the instructions are executed by the processor, the efficient data sampling and processing method of embodiment 1 is implemented.

[0067] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. An efficient data sampling and processing method, characterized in that: The following steps are involved: Create parallel data reading and writing threads and data processing threads, configure the circular cache queue, flip cache queue and the reading recorder of each channel; each channel is preset with a unique reading recorder to control the update logic of the channel's cache data. The reading recorder is used to record the queue length of the channel cache data queue, the length of the written data, the remaining updateable data length and the current measurement position; The flip cache queue includes a forward cache area and a backward cache area. The forward cache area is used by the data reading and writing thread to write the newly collected data, while the backward cache area is used for reading by the data processing thread. The processing process of the data reading and writing thread includes: cyclically reading the instant sampling data from the hardware memory and writing it into the circular buffer queue in order, reading the instant sampling data from the circular buffer queue into the channel buffer data queue in turn according to the reading recorder information of each channel, and updating the buffer data in the specified channel buffer data queue, pushing the updated buffer data to the forward buffer area of ​​the flip buffer queue, and waiting for the data processing thread to process it; The processing process of the data processing thread includes: cyclically reading updated cache data from the backward cache area in the flip cache queue, flipping the forward cache area and the backward cache area, and processing the read cache data.

2. The efficient data sampling and processing method according to claim 1, characterized in that: The process of configuring a circular cache queue includes setting the queue's head pointer, tail pointer, start pointer, and total buffer length, monitoring the position status of the head pointer and tail pointer in real time, and calculating the cached length and the remaining writable length. Before writing the instant sampling data, the length of the currently unprocessed sampling data is calculated based on the head pointer and tail pointer to obtain the cached length, and it is determined whether the cached length exceeds a preset threshold. If so, it is determined that there is redundant data in the circular cache queue, the latest instant sampling data is retained, the redundant data in the circular cache queue is erased, and the position status of the head pointer and tail pointer, the cached length, and the remaining writable length are updated.

3. The efficient data sampling and processing method according to claim 2, characterized in that: The specific process of writing sampled data into the ring buffer queue includes: According to the position of the tail pointer, the remaining writable length of the ring buffer queue is calculated. If the length of the real-time sampling data that can be read in the hardware memory is less than the remaining writable length, the real-time sampling data is written into the ring buffer queue, and the tail pointer position is refreshed to the position after writing; If the length of the real-time sampling data that can be read in the hardware memory exceeds the remaining writable length, the real-time sampling data is divided into two parts and written, one part is written to the buffer area corresponding to the remaining writable length, and the other part of the data is written from the beginning of the buffer area corresponding to the start pointer, and the tail pointer position is refreshed to the position after writing; After writing the instant sampling data, the position status of the head pointer and tail pointer of the circular buffer queue, the buffered length, and the remaining writable length are updated.

4. The efficient data sampling and processing method according to claim 2, characterized in that: The preset threshold is three times the total length of the channel cache data queues of all channels.

5. The efficient data sampling and processing method according to claim 1, characterized in that: The process of reading instant sampled data from the ring buffer queue, updating the buffer data of the specified channel and writing it to the flip buffer queue is as follows: The data reading and writing thread cyclically obtains the instant sampling data from the ring buffer queue and writes it into the channel buffer data queue of each channel in turn. For each channel, the update process is: Get the remaining updateable data length and current measurement position activated in the reading recorder of the current writing channel; The instantaneous sampling data of the remaining updateable data length is read from the circular buffer queue, and the instantaneous sampling data is sequentially written into the channel buffer data queue of the current channel starting from the current measurement position, and the information recorded by the current reading recorder is refreshed.

6. The efficient data sampling and processing method according to claim 1, characterized in that: The channel cache data queue is a queue of fixed length, and the queue length corresponds to the data length of the forward and backward cache areas in the flip cache queue respectively.

7. The efficient data sampling and processing method according to claim 1, characterized in that: When updating the designated channel cache data queue, the amount of data that can be updated in the channel cache data queue is calculated based on the current measurement position, the length of the written data and the remaining updateable data length of the matched reading recorder, combined with the total amount of data corresponding to the cached length in the ring cache queue; If the cached length in the ring buffer queue is less than the remaining updateable data length, all data corresponding to the cached length in the ring buffer queue are stored in the specified channel buffer data queue; If the cached length in the ring buffer queue is greater than the remaining updateable data length, only the amount of data corresponding to the remaining updateable data length is read and stored in the specified channel buffer data queue.

8. The efficient data sampling and processing method according to claim 1, characterized in that: Writing to the forward buffer and reading from the backward buffer are controlled by the same thread lock.

9. A computer device, characterized in that: include: at least one processor; and a memory, wherein the memory stores computer instructions that can be run on the processor, and when the instructions are executed by the processor, an efficient data sampling and processing method according to any one of claims 1 to 8 is implemented.

Images