Real-time high-frequency sensor data acquisition system

Abstract

The invention provides a real-time high-frequency sensor data acquisition system, which comprises an acquirer firmware program and an upper computer software program, and is characterized in that the acquirer firmware program acquires sensor data according to set integral time, encapsulates data acquired in a single or segmented manner into a private message format, and sends the data to an upper computer in real time through a communication module. The upper computer software is provided with a high-priority communication receiving thread and is used for receiving the segmented data messages, splicing the segmented data messages into complete frame data and storing the complete frame data into a first-level cache region; the system further comprises a plurality of data processing threads and corresponding second-level cache regions, and the data processing threads are used for processing frame data in parallel. The system supports sensors in different acquisition modes, has high-frequency data acquisition capability, improves data processing efficiency through dynamic thread scheduling, and solves the problems of data frame loss and incapability of real-time processing in a traditional architecture.

Description

Technical Field

[0001] The present invention relates to an acquisition system, in particular to a real-time high-frequency sensor data acquisition system. Background Art

[0002] In modern industrial systems, real-time data collection is a core component of intelligent manufacturing, equipment monitoring, and production process optimization. By collecting real-time data on equipment operating status, environmental parameters, production indicators, and other data, companies can achieve precise monitoring, predictive maintenance, and efficient decision-making. For example, in synthetic biology, data acquisition systems are used to monitor key parameters of fermenters and bioreactors in real time, such as temperature, pH, dissolved oxygen, cell density, and fermentation broth concentration. Combined with edge computing, they analyze microbial metabolic states, optimize culture conditions, and increase product yield. In fruit and vegetable sorting, data acquisition systems use near-infrared spectroscopy (NIRS) technology to monitor moisture content, sugar content, and mold in real time, improving sorting accuracy. In coal quality monitoring, data acquisition systems integrate equipment such as belt scales, gamma-ray ash analyzers, and laser-induced breakdown spectroscopy (LIBS) to collect real-time data on coal flow, including weight, ash content, sulfur content, and calorific value, to control the quality of selected coal. Therefore, ensuring the accuracy, real-time nature, and stability of data acquisition is crucial to improving the efficiency and reliability of industrial production.

[0003] Although data acquisition technology has been widely used in the industrial field, existing solutions still have several key issues, which limit their application in high-demand scenarios.

[0004] (1) Frame loss often occurs when the host computer software collects data. Many traditional data acquisition systems rely on host computer software for data processing and storage. However, due to computer performance limitations, operating system scheduling delays, or insufficient communication bandwidth, data is prone to frame loss during transmission. For example, when a sensor sends high-frequency data while the host computer is processing other tasks, the host computer may lose some data packets due to insufficient processing power, resulting in incomplete collection results and affecting subsequent analysis and control decisions.

[0005] (2) Unable to adapt to scenarios requiring high-frequency data acquisition. In applications such as vibration analysis and high-speed motion control, data acquisition frequencies must reach the kilohertz (KHz) or even megahertz (MHz) level. However, existing data acquisition systems are often limited by the hardware performance of the host computer and the unreasonable architecture of the software data acquisition, making it difficult to meet high-frequency acquisition requirements. This may also result in data delays or loss due to the inability to guarantee real-time performance due to the unreasonable software acquisition architecture.

[0006] (3) It is not compatible with all data acquisition sensor types. The data acquisition methods of sensors are mainly divided into two modes: single-shot acquisition of one frame of data and multiple-segment acquisition of one frame of data, which depends on the design of the sensor, bandwidth limitations or application requirements. For example, industrial cameras usually support single exposure to output a complete frame of image (such as Hamamatsu's S11639CMOS linear array sensor), which is suitable for high-speed pipeline detection and has strong real-time data. However, some high-resolution or large dynamic range sensors (such as CCD linear array sensors used in spectrometers) may be limited by ADC conversion speed or data transmission bandwidth, and need to be scanned multiple times and then spliced ​​into a complete frame of data. For example, Hamamatsu's S11156 back-illuminated CCD may use multiple integrations to improve the signal-to-noise ratio when detecting weak light. In addition, in X-ray detection, large-area detectors (such as C13340 linear array sensors) may also scan objects line by line to finally synthesize a complete image. For the mode of multiple-segment acquisition of one frame of data, the existing technical solutions cannot achieve high-frequency data acquisition.

[0007] These issues not only reduce the reliability of data collection but also limit the application of industrial systems in high-speed, high-precision scenarios. Therefore, a more efficient, flexible, and stable data collection solution is urgently needed.

[0008] Existing technical solutions for collecting sensor data generally transmit raw data without a proprietary communication protocol message design. Each transmitted data is a complete frame of data. The data is directly received using the operating system's synchronization notification and the frame data is directly processed in the system callback interface. The technical process of the existing technical solution is as follows: (1) First, start the collector firmware program and set the integration time for collecting sensor data; the firmware program collects complete frame data according to the integration time. If the sensor does not support the collection of complete frame data in a single time, the data is segmented multiple times and cached in the firmware program until the complete frame data is collected. The communication firmware module is directly called to send it to the host computer program.

[0009] (2) After the host computer software starts, it immediately registers a callback notification interface function with the operating system's communication port. After the operating system's communication port receives the original data message sent by the lower computer firmware program, it immediately notifies the execution of the previously registered callback interface function. Every time the operating system receives a frame of data message, it notifies the execution of the callback interface function.

[0010] (3) In the callback interface function, the frame data message is directly calculated and processed; if the user needs the detection function, the frame data is detected and calculated in the function interface; if the user needs to display the detection results in real time, the detection results are written to the buffer area in the function interface; if the user needs to save the frame data, the frame data is written to the disk file in the function interface.

[0011] (4) The upper computer software program creates a user operation interface, directly reads the detection calculation result data from the buffer area, and displays it in real time in the software interface.

[0012] The process of the existing technical solution for collecting sensor data is as follows: Figure 1 shown.

[0013] Existing technical solutions for collecting sensor data generally transmit raw data without a proprietary communication protocol message design. Each transmitted data is a complete frame of data. The data is directly received using the operating system's synchronization notification and the frame data is directly calculated and processed in the system callback interface. The disadvantages are as follows: 1. When the host computer software collects data, frame loss often occurs. This can be caused by jitter in the communication interface transmission efficiency or performance limitations of the computer where the host computer software is installed.

[0014] 2. Unsuitable for high-frequency data acquisition scenarios. The data processing logic is directly embedded in the callback interface function registered with the operating system communication port. The host computer's data receiving and acquisition logic and calculation and processing logic cannot be carried out in parallel, which will slow down the data acquisition rate and cannot meet the needs of high-frequency data acquisition scenarios.

[0015] 3. Incompatibility with all data acquisition sensor types. Sensor data acquisition methods are mainly divided into two modes: single-shot acquisition of a frame of data and multiple segmented acquisition of a frame of data. For the multiple segmented acquisition of a frame of data mode, existing technical solutions cannot achieve high-frequency data acquisition. Summary of the Invention

[0016] In order to solve the above problems, the present invention provides a real-time and high-frequency sensor data acquisition system, and designs a private message format that supports segmented data transmission. The firmware program collects a segment of data and immediately assembles it into a package and sends it to the host computer software; the host computer creates an independent thread for receiving communication data and sets it as a high-priority thread. This thread is only used to receive communication data and write the data into a data buffer; the frame data is calculated and processed in the data processing thread, and the number of data processing threads can be increased or decreased according to actual calculation and processing needs.

[0017] The present invention is achieved through the following technical solutions: A real-time high-frequency sensor data acquisition system, comprising: The collector firmware program is set in the collector to control the data acquisition of the sensor and collect a frame of data according to the set integration time, where: When the sensor supports single-shot acquisition of full-frame data, the full-frame data is acquired directly; When the sensor needs to collect a frame of data in multiple segments, the collector firmware program collects each segment of data and encapsulates it into a segmented message; The segmented message includes a frame header, message type, frame number, segment number, data length, original data and a frame tail checksum; After the collector firmware program collects a piece of data, it immediately sends the encapsulated segmented message to the host computer software through the communication module; The host computer software program runs on the computer device and includes: A high-priority communication receiving thread for receiving segmented messages. This thread does not rely on the operating system callback mechanism and is only used to continuously receive communication data and store the data in the first-level cache; A message reassembly module for splicing multiple segmented data in the first-level buffer into complete frame data according to the frame number and segment number; At least one data processing thread, each data processing thread corresponding to a secondary cache block, for reading the spliced ​​frame data from the primary cache block and performing subsequent processing operations; The data processing thread management module is used to dynamically increase or decrease the number of data processing threads and the corresponding L2 cache blocks according to the processing load to adapt to data processing requirements of different frequencies; Among them, the communication message adopts a private protocol format, supports compatibility with multiple types of sensor acquisition modes, and the system as a whole has real-time high-frequency data acquisition capabilities and supports dynamic thread scheduling to improve processing efficiency.

[0018] As a preferred technical solution, the frame header of the segmented message is 4 bytes and is fixed to AABBCCDD. The frame tail checksum is obtained by accumulating the two bytes of each field of the above message and taking the lower two bytes.

[0019] As a preferred technical solution, the priority of the communication receiving thread is higher than the priority of the operation interface thread and the data processing thread, and it runs continuously to avoid message loss due to interface or calculation blocking.

[0020] As a preferred technical solution, the data processing thread management module triggers thread adjustment in the following situations: When the number of frames to be processed in all L2 caches is greater than or equal to 2, a data processing thread and the corresponding L2 cache block are automatically added; When the number of frames to be processed in all L2 caches is less than or equal to 1, a data processing thread and its L2 cache block are automatically reduced.

[0021] As a preferred technical solution, after completing the splicing of a frame of data, the splicing module compares the total time taken to receive and splice the frame with the set integration time. If it is less than the set time, a waiting delay operation is performed to match the sampling period.

[0022] As a preferred technical solution, the data processing thread can perform user-defined logical operations including but not limited to data analysis, visualization conversion, storage writing, and alarm judgment when processing frame data.

[0023] As an optimal technical solution, after receiving the segmented data message, the communication receiving thread in the host computer software sorts and verifies it according to the segment number to ensure that each segment of data is spliced ​​into complete frame data in the correct order to avoid frame data confusion or repeated splicing.

[0024] As a preferred technical solution, the collector firmware program dynamically adjusts the data collection time based on the current system state before sending each segmented data message to match the response time of different types of sensors and the data frame construction requirements.

[0025] As a preferred technical solution, after the frame data is spliced ​​and written into the first-level cache, it will be accompanied by an integrity mark field to indicate whether the current frame data is spliced ​​completely, so that the main thread of data processing can judge its availability.

[0026] As an optimal technical solution, the host computer software includes a parameter configuration module for setting the sensor integration time, sampling frequency, thread number upper limit and cache threshold. The parameter configuration module and the collector firmware program synchronize parameters through communication messages.

[0027] The beneficial effects of the present invention are as follows: First, the present invention introduces a mechanism that supports segmented data transmission in the collector firmware and sets a high-priority independent thread in the host computer software to receive communication data, effectively bypassing the operating system scheduling delay and processing bottleneck, significantly reducing the risk of frame loss during data reception, and ensuring data transmission integrity; Second, the present invention supports both single-shot acquisition of a complete frame of data and multiple segmented acquisition of a frame of data. It is compatible with sensor devices with different acquisition principles and bandwidth limitations, such as linear array CCDs, CMOS, spectrometers, and X-ray detectors, and has a wider range of application scenarios. Third, by separating the communication data reception and data processing processes and adopting an independent thread mechanism and cache management, the performance bottleneck of synchronous processing in traditional architectures is broken, effectively supporting high-frequency data acquisition tasks at the kHz and even MHz levels, and is suitable for industrial scenarios with high real-time requirements such as vibration monitoring and motion control. 4. The host computer supports dynamic adjustment of the number of data processing sub-threads based on actual data processing pressure, and cooperates with the cache load sensing mechanism to ensure data processing efficiency while reducing resource usage, thereby improving the overall stability and scalability of system operation; 5. The message structure adopts a private protocol, including fields such as frame header, message type, frame number, segment number, data length and frame tail checksum, which is conducive to rapid data parsing and error detection, and improves the robustness and security of the communication protocol; 6. The host computer can set the integration time, sampling frequency, number of threads and other operating parameters through communication messages, and synchronize with the collector firmware in real time to enhance the system's configurability and user controllability, making it easier to deploy and maintain. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 It is a system block diagram in the prior art; Figure 2 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0030] All features disclosed in this specification, or all steps in the disclosed methods or processes, except mutually exclusive features and / or steps, can be combined in any manner.

[0031] Any feature disclosed in this specification (including any appended claims, abstract, and drawings), unless otherwise stated, may be replaced by other equivalent or similar features. In other words, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0032] like Figure 2 As shown, the present invention proposes a real-time high-frequency sensor data acquisition system, and its specific implementation is as follows: First, a data collector hardware platform is provided. Upon power-up, the platform automatically starts the embedded data collector firmware. The data collector collects data by connecting to various external sensors (such as industrial cameras, spectrometers, CCDs, CMOS linear arrays, and X-ray detectors). According to the configuration instructions sent by the host computer, the collector firmware program sets the sensor integration time, thereby determining the acquisition time of each frame of data; Depending on its own characteristics, the sensor may support a single acquisition of a complete frame of data, or it may need to collect multiple segments of data multiple times and then assemble them into a complete frame of data due to resolution or bandwidth limitations.

[0033] After collecting a complete frame of data or each segment of data, the data collector firmware immediately encapsulates it into a communication message with a specific format. This communication message uses a proprietary protocol and includes the following format: frame header (4 bytes), message type (1 byte), frame number (2 bytes), segment number (1 byte), data length (2 bytes), raw data (n bytes), and a checksum at the end of the frame (2 bytes). The frame header is fixed at AABBCCDD and is used to identify the start of the message; the message type indicates the data type, such as frame data, heartbeat signal, or temperature information; the frame number is used to distinguish the order of data frames; the segment number marks the order of each segment of the current frame; 0 indicates no segmentation is required; the data length indicates the number of bytes in the subsequent data body; and the checksum at the end of the frame is the two lower digits of the two-byte cumulative value of the aforementioned fields and is used to verify message integrity.

[0034] In terms of firmware logic, if the sensor supports single-shot acquisition of a complete frame of data, the collector directly encapsulates the entire frame into a message and sets the segment number to 0. If the sensor is in segmented output mode, a message is constructed for each segment of data acquired, the current segment number is set, and the message is then sent to the host computer software system via the communication module. This process ensures the collector's flexible compatibility with different sensor types and enables real-time transmission of collected data.

[0035] In the host computer software, the system first creates a high-priority, independent communication receiving thread. This thread establishes a stable connection with the firmware's communication module and is dedicated to receiving segmented data packets from the data collector. This thread does not rely on the synchronous callback mechanism of the operating system's serial / USB port. Instead, it always runs in an active listening state to ensure that message reception during the high-speed data collection process is not interrupted by other system tasks, effectively avoiding message loss caused by operating system scheduling or insufficient computing resources.

[0036] After receiving each message, the communication receiving thread will parse its segment number and insert the data into the corresponding frame buffer structure according to the frame number and segment number. When all segments are received and spliced ​​into a complete frame data, the frame data is written to the first-level buffer area for processing; After receiving a complete frame, the system calculates the time required for data stitching and compares it with the configured integration time. If the processing time is shorter than the integration time, a waiting delay is performed before starting the next round of acquisition. If it exceeds the integration time, the system immediately enters the next round of data stitching processing. This mechanism ensures the stability and timing consistency of the data acquisition cycle.

[0037] The data processing portion is scheduled by a high-priority main data processing thread. This thread reads complete frames of data from the L1 cache and distributes them to multiple data processing sub-threads in a round-robin or idle-first manner. Each sub-thread is equipped with an independent L2 cache block for storing frames of data to be processed. The data processing sub-thread is responsible for performing computations on the frame data, such as image analysis, spectrum analysis, signal determination, result display, or storage and disk writing. All computational processing threads operate independently and do not interfere with each other.

[0038] The system further includes a dynamic thread management mechanism. If the number of pending frames in the L2 cache of all processing threads is greater than or equal to 2, the main thread automatically triggers the addition of a new processing sub-thread and its cache. If the number of pending frames in the cache of all processing threads is less than or equal to 1, the main thread proactively releases redundant sub-threads and their cache resources, thereby achieving dynamic resource balancing and optimal utilization. This mechanism can expand or contract the number of threads in real time based on sampling rate and processing requirements, effectively preventing processing bottlenecks and improving system stability under varying loads.

[0039] In addition to the basic communication, caching, and processing mechanisms, the system of the present invention further supports flexible parameter configuration through the host computer software. The system provides a parameter configuration module that allows users to set key parameters such as integration time, sampling frequency, maximum number of processing threads, cache queue threshold, etc. according to different sensors or field requirements; The module interacts with host computer software via a graphical user interface or command interface. Setting results are transmitted to the data collector firmware via communication messages, enabling synchronized parameter updates between the host and host computers. This mechanism allows users to flexibly address the differentiated data collection requirements of different sensor types in real-world applications, enhancing system controllability and versatility.

[0040] During system operation, the three core modules work in parallel and stably. First, the collector firmware continuously collects segmented data and encapsulates the data into messages for transmission in real time. Second, the host computer communication receiving thread runs continuously to ensure that the segmented messages are fully received and spliced ​​into complete data frames and written to the first-level buffer. Finally, the main data processing thread continuously reads complete data frames from the first-level buffer and distributes them to various sub-threads for processing, while dynamically adjusting the number of threads to match changes in data load.

[0041] The entire collection, reception, and processing process continues to run from the start until the user actively terminates the collection operation through interface operation or external command; The termination command is sent by the host computer software to the collector. After receiving the command, the collector firmware shuts down the sensor acquisition logic and stops sending messages. The host computer software also automatically stops the receiving thread and data processing thread and releases related cache resources. The entire system realizes a complete, continuous and reliable data acquisition closed-loop process.

[0042] Through the above technical solutions, the present invention has achieved several key performance breakthroughs: First, the system significantly improves its adaptability to high-frequency data acquisition tasks, solving the problem of frequent frame drops caused by processing blockage in traditional synchronous callback processing architectures. Second, the private message protocol structure is well designed, supporting segmented data identification, sorting, splicing, and integrity verification, and has good scalability and communication stability. Third, an efficient communication link is established between the collector firmware and the host computer software through independent threads and a cache mechanism, ensuring that the host computer can reliably complete high-speed data reception even in a multi-tasking environment; Fourth, the data processing part adopts a dynamic thread scheduling mechanism, which can adjust the number of processing resources in real time according to actual load changes, taking into account both processing efficiency and system stability; Fifth, the system has good compatibility and supports two mainstream sensor acquisition modes: single acquisition and multiple segmented acquisition. It is suitable for various industrial real-time monitoring scenarios such as bioreactor monitoring, fruit and vegetable sorting, and coal quality testing.

[0043] The real-time high-frequency sensor data acquisition system proposed in the present invention systematically solves the core problems of traditional data acquisition systems in terms of high frequency, high reliability and diversified sensor support through message protocol optimization, thread architecture transformation and cache mechanism collaborative design. It has significant engineering application value and industrial promotion prospects.

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that are not conceived through creative work should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection defined in the claims.

Claims

1. A real-time high-frequency sensor data acquisition system, characterized in that: include: The collector firmware program is set in the collector to control the data acquisition of the sensor and collect a frame of data according to the set integration time, where: When the sensor supports single-shot acquisition of full-frame data, the full-frame data is acquired directly; When the sensor needs to collect a frame of data in multiple segments, the collector firmware program collects each segment of data and encapsulates it into a segmented message; The segmented message includes a frame header, message type, frame number, segment number, data length, original data and a frame tail checksum; After the collector firmware program collects a piece of data, it immediately sends the encapsulated segmented message to the host computer software through the communication module; The host computer software program runs on the computer device and includes: A high-priority communication receiving thread for receiving segmented messages. This thread does not rely on the operating system callback mechanism and is only used to continuously receive communication data and store the data in the first-level cache; A message reassembly module for splicing multiple segmented data in the first-level buffer into complete frame data according to the frame number and segment number; At least one data processing thread, each data processing thread corresponding to a secondary cache block, for reading the spliced ​​frame data from the primary cache block and performing subsequent processing operations; The data processing thread management module is used to dynamically increase or decrease the number of data processing threads and the corresponding L2 cache blocks according to the processing load to adapt to data processing requirements of different frequencies; Among them, the communication message adopts a private protocol format, supports compatibility with multiple types of sensor acquisition modes, and the system as a whole has real-time high-frequency data acquisition capabilities and supports dynamic thread scheduling to improve processing efficiency.

2. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: The frame header of the segmented message is 4 bytes and is fixed to AABBCCDD. The frame tail checksum is obtained by accumulating the two bytes of each field of the above message and taking the lower two bytes.

3. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: The communication receiving thread has a higher priority than the operation interface thread and the data processing thread, and runs continuously to avoid message loss due to interface or calculation blocking.

4. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: The data processing thread management module triggers thread adjustment in the following situations: When the number of frames to be processed in all L2 caches is greater than or equal to 2, a data processing thread and the corresponding L2 cache block are automatically added; When the number of frames to be processed in all L2 caches is less than or equal to 1, a data processing thread and its L2 cache block are automatically reduced.

5. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: After completing the splicing of a frame of data, the splicing module compares the total time consumed for receiving and splicing the frame with the set integration time. If the total time is less than the set time, a waiting delay operation is performed to match the sampling period.

6. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: When processing frame data, the data processing thread can perform user-defined logical operations including but not limited to data analysis, visualization conversion, storage writing, and alarm judgment.

7. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: After receiving the segmented data message, the communication receiving thread in the host computer software sorts and verifies it according to the segment number to ensure that each segment of data is spliced ​​into complete frame data in the correct order to avoid frame data confusion or repeated splicing.

8. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: The collector firmware program dynamically adjusts the data collection time based on the current system state before sending each segmented data message to match the response time of different types of sensors and the data frame construction requirements.

9. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: After the frame data is spliced ​​and written into the primary cache, it will be accompanied by an integrity mark field to indicate whether the current frame data is spliced ​​completely, so that the main thread of data processing can judge its availability.

10. The real-time high-frequency sensor data acquisition system according to claim 1, characterized in that: The host computer software includes a parameter configuration module for setting the sensor integration time, sampling frequency, thread number upper limit and cache threshold. The parameter configuration module and the collector firmware program synchronize parameters through communication messages.

Images