A data acquisition method and system based on lock-free buffer

Abstract

The application discloses a kind of data acquisition method and system based on lockless buffer zone, the system includes: data acquisition module, for the data required by system is collected, is transmitted by network module;Data register, adopt ring chain data structure, be provided with lockless data buffer zone, wherein data storage area is used to temporarily store the data required by system from data acquisition module, data management area is used to classify management of the data of data storage area;Conveying thread module, for the data required by system in data register is conveyed to database in the way of multithread concurrent execution;Database, adopt nested linked list structure, for persisting the data conveyed by conveying thread module;Monitoring module, for deleting after identifying to invalid data in the data required by system.The application can quickly convey the acquisition data to database and persist, for invalid data can also be timely reduced, improve the speed of information processing.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and specifically to a data acquisition method and system based on a lock-free buffer. Background Technology

[0002] With the rapid development of science and technology, existing information systems are usually implemented using pure software. As a result, the performance bottleneck problem is becoming increasingly prominent. In the process of information collection and processing, the main problem is that after the data packets enter the backend server, the data cannot be cached in a timely and effective manner, resulting in processing delays, or even instability and paralysis of the data system, thereby reducing the efficiency of information processing and increasing the cost of information processing.

[0003] Therefore, how to efficiently process the collected data has become an urgent problem for the entire society. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a data acquisition method and system based on a lock-free buffer to improve information processing efficiency.

[0005] In a first aspect, embodiments of the present invention provide a data acquisition system based on a lock-free buffer, the system comprising:

[0006] The data acquisition module is used to collect the data required by the system and transmit it to the data buffer through the network module.

[0007] The data buffer adopts a ring-linked data structure and is equipped with a lock-free data buffer, wherein: the data storage area in the lock-free data buffer is used to temporarily store the system-required data from the data acquisition module; the data management area in the lock-free data buffer is used to classify and manage the data in the data storage area.

[0008] The transmission thread module is used to transmit the system's required data, which is temporarily stored in the data buffer, to the database in a multi-threaded concurrent execution manner.

[0009] The database uses a nested linked list structure to persistently store the data transmitted by the transmission thread module.

[0010] The monitoring module is used to identify invalid data in the system's required data that does not meet the set requirements, and to delete the invalid data.

[0011] Furthermore, the monitoring module is used to identify invalid data in the system's required data that does not meet the set requirements, and delete the invalid data, which may include:

[0012] Monitor the system's required data stored in the monitoring data buffer and identify invalid data that does not meet the set requirements;

[0013] Delete the identified invalid data stored in the database.

[0014] Furthermore, the monitoring module is used to monitor the system-required data temporarily stored in the data buffer and identify invalid data that does not meet the set requirements, and may include:

[0015] Monitor the data in the outflow buffer and invalidation buffer within the lock-free data buffer;

[0016] The data in the overflow buffer and invalid buffer will be invalid data that does not meet the set requirements.

[0017] Furthermore, the lock-free data buffer creation setting includes two groups, each group of lock-free data buffers including a data storage area and a data management area;

[0018] The data buffer is used to alternately store and classify the data required by the system, using two lock-free data buffers.

[0019] Furthermore, the system also includes a data processing module for synchronizing the data to be synchronized in the lock-free data buffer.

[0020] Secondly, embodiments of the present invention also provide a data acquisition method based on a lock-free buffer, the method comprising:

[0021] Collect the data required by the system;

[0022] The system temporarily stores the collected data in the data storage area of ​​the lock-free data buffer, and the data management area in the buffer is used to classify and manage the data in the data storage area.

[0023] The system uses a multi-threaded concurrent execution method to transfer temporarily stored system data to the database.

[0024] The database uses a nested linked list structure to persistently store the data required by the system.

[0025] Identify invalid data in the data required by the system that does not meet the set requirements, and delete the invalid data.

[0026] Furthermore, invalid data that does not meet the set requirements in the data required by the system is identified and deleted, including:

[0027] Monitor the system's required data temporarily stored in the lockless data buffer and identify invalid data that does not meet the set requirements;

[0028] Delete the identified invalid data stored in the database.

[0029] Furthermore, monitor the system's required data temporarily stored in the lock-free data buffer, and identify invalid data that does not meet the set requirements, including:

[0030] Monitor the data in the overflow buffer and invalid buffer within the lock-free data buffer;

[0031] The data in the overflow buffer and invalid buffer will be invalid data that does not meet the set requirements.

[0032] Furthermore, two sets of lock-free data buffers are provided, each set including a data storage area and a data management area;

[0033] Two sets of lock-free data buffers are used alternately to temporarily store and categorize the data required by the system.

[0034] Furthermore, the data acquisition method also includes: performing synchronization processing on the data to be synchronized in the lock-free data buffer.

[0035] The technical solution provided by this invention can temporarily store system data in a lock-free data buffer for classification and management after collection. Subsequently, it can be quickly transmitted to the database for persistent storage in a multi-threaded concurrent execution manner. Invalid data can also be deleted in a timely manner, which greatly improves the speed of information processing. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of a data acquisition system based on a lock-free buffer provided in an embodiment of the present invention;

[0037] Figure 2 This is a flowchart illustrating a data acquisition method based on a lock-free buffer according to the present invention. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the accompanying drawings.

[0039] First, this invention provides a data acquisition system based on a lock-free buffer, see [link to relevant documentation]. Figure 1 The system includes: a data acquisition module 101, a network module 102, a data buffer 103, a transmission thread module 104, a database 105, and a monitoring module 106. The functions of each module are described in detail below.

[0040] (1) Data acquisition module 101

[0041] This module is used to collect the data required by the system and transmit it to the data buffer via network module 102. The collected system data can include various types, depending on the system's design and purpose. Generally, the system data includes, but is not limited to, the following categories:

[0042] User information: This includes basic user information such as name, gender, age, and contact information;

[0043] Behavioral data: This includes user actions within the system, such as clicks, browsing, purchases, and comments.

[0044] Device information: This includes information about the devices used by the user, such as device model, operating system version, network type, etc.

[0045] Location information: This includes the user's geographical location, which can be obtained through GPS positioning or IP address, etc.

[0046] Log data: includes log records generated during system operation, used for system troubleshooting and performance optimization;

[0047] Text data: This includes text data entered by users into the system, such as search keywords and comment content;

[0048] Image or audio data: including user-uploaded image or audio files, such as user avatars, voice messages, etc.;

[0049] Third-party data: The system may interact with other applications or services to obtain third-party data for the implementation of system functions.

[0050] Furthermore, the network module 102 can be a wired network or a wireless network (such as 2G, 3G, 4G, 5G, WIFI, Bluetooth, etc.), preferably using WIFI technology.

[0051] (2) Data temporary register 103

[0052] This module includes a lock-free data buffer. In this buffer, one or more atomic operations are used to read and write data, instead of using mutexes to protect access to shared data. This avoids thread contention and blocking, improving concurrency performance and response speed. There are many ways to implement a lock-free data buffer, including lock-free queues (such as lock-free circular buffers, lock-free linked lists, etc.) and lock-free hash tables. Preferably, a circular linked data structure is used. This structure achieves thread-safe concurrent read and write operations through atomic operations and specific algorithms, thus providing an efficient data transmission and sharing mechanism.

[0053] For example, the lock-free data buffer includes two areas: a data storage area and a data management area. The data storage area is used to temporarily store system-required data from the data acquisition module 101; the data management area is used to classify and manage the data in the data storage area. In specific implementations, the system-required data can be classified according to its source, for example, into:

[0054] Manually entered data: Data entered manually by the user, such as data entered through a form or text editor;

[0055] Sensor data: Data collected by various sensors, such as temperature, humidity, and pressure sensors;

[0056] Network data: Data obtained through network interfaces, such as data returned by API interfaces, data crawled from web pages, etc.

[0057] Alternatively, the data required by the system can be categorized according to its intended use, for example, into:

[0058] Monitoring data: Used to monitor and detect system status, such as network monitoring data, equipment operating status, etc.

[0059] Data analysis: Used for data analysis and mining, such as statistical analysis and machine learning;

[0060] Decision data: used to support decision-making and management, such as business data, market data, etc.

[0061] Visualized data: Used for data visualization, such as charts and reports.

[0062] Of course, the data required by the system can also be classified according to the business type, depending on specific needs and application scenarios. This embodiment of the invention does not impose specific limitations on the classification method.

[0063] In a preferred embodiment, the lock-free data buffer is configured in two sets, each set including a data storage area and a data management area. The data buffer 103 is used to alternately utilize the two lock-free data buffers to temporarily store and categorize the data required by the system. While one lock-free buffer is being read, the other can be written to simultaneously, thereby reducing the waiting time between read and write operations and improving the system's processing speed and responsiveness.

[0064] (3) Transmission Thread Module 104

[0065] This module is used to transfer the system-required data temporarily stored in the data register 103 to the database 105 in a multi-threaded concurrent execution manner.

[0066] (4) Database 105

[0067] This module uses a nested linked list structure to persistently store the data transmitted by the transmission thread module 104.

[0068] (5) Monitoring module 106

[0069] This module is used to identify invalid data in the system's required data that does not meet the set requirements, and to delete the invalid data.

[0070] For example, the monitoring module 106 is specifically used to monitor the system-required data temporarily stored in the data buffer 103, identify invalid data that does not meet the set requirements, and delete the identified invalid data stored in the database 105.

[0071] In practical applications, overflow buffers and invalid buffers frequently occur in the lock-free data buffer of data temporary register 103. Theoretically, data in the overflow buffer can be stored in database 105. An overflow buffer occurs when data is written to a buffer exceeding its capacity, potentially causing data to overflow into an adjacent memory area. If this adjacent memory area is a storage area of ​​database 105, the data in the overflow buffer might be stored in database 105. However, it's important to note that overflow buffers are usually a programming or coding error, not an intentional data storage method. Data in the overflow buffer can lead to program crashes, security vulnerabilities, etc. Therefore, directly storing data in the overflow buffer in database 105 is inappropriate. Furthermore, since the invalid buffer itself may be invalid or unavailable, data may not be correctly stored in database 105. Therefore, in this embodiment of the invention, as a specific implementation, the data in both the overflow buffer and the invalid buffer in the lock-free data buffer are considered invalid and deleted. Correspondingly, the monitoring module 106 monitors the system-required data temporarily stored in the data buffer 103, identifying invalid data that does not meet the set requirements. Specifically, this includes monitoring the data in the overflow buffer and invalid buffer within the unlocked data buffer; and classifying the data in the overflow buffer and invalid buffer as invalid data that does not meet the set requirements. If the system obtains the required data from the database 105 for subsequent processing, only the identified invalid data stored in the database 105 can be deleted. The deleted data may be all or part of the identified invalid data. Optionally, the identified invalid data stored in the data buffer 103 can be deleted. Furthermore, the deleted invalid data can be collected a second time, cached, and then persistently stored until it is correctly stored in the database 105.

[0072] Of course, invalid data that does not meet the set requirements can also be identified through other methods. Specifically, this can be determined based on the system's business needs, including:

[0073] Data range check: The range of the collected system data is checked to determine whether it exceeds a reasonable value range. For example, when the temperature sensor collects data, a reasonable temperature range can be set. If the collected system data exceeds the range, it is determined to be invalid data.

[0074] Data anomaly detection: Using statistical analysis methods or machine learning algorithms, anomalies are detected in the collected system data; if the data differs significantly from normal patterns or historical data, it can be considered invalid data; for example, the presence of abnormal data can be determined by monitoring the fluctuation or trend of sensor data.

[0075] Business rule validation: Based on preset business rules and logic, determine whether the collected system data is invalid; for example, when collecting user registration information, it can determine whether the username is unique and whether the password meets security requirements.

[0076] Data quality assessment: By using preset data quality assessment indicators, the collected data required by the system is evaluated. For example, the accuracy, consistency, completeness, and reliability of the data can be assessed, and low-quality data can be judged as invalid data.

[0077] It is important to note that the methods for determining invalid data may differ depending on the specific business scenario. Therefore, in practical applications, it is necessary to combine the above-mentioned methods in combination with the specific circumstances to determine the appropriate approach. Typically, for methods that identify invalid data based on system business requirements, invalid data can be deleted from the data temporary storage 103 before it is transmitted to database 105. The advantages of this approach are:

[0078] Ensuring database data quality: Deleting invalid data ensures that the database contains only valid and accurate data; invalid data may be generated due to errors, anomalies, or other reasons, and deleting it can prevent it from affecting the data quality of the database.

[0079] Save storage space: Deleting invalid data in a timely manner can free up storage space in data buffers and databases, improving the utilization rate of storage resources;

[0080] Improving data processing efficiency: Invalid data may consume data processing time and resources; if invalid data is passed into the database for processing, it will increase the database load and may prolong data processing time; timely deletion of invalid data can reduce the workload of data processing and improve data processing efficiency and performance.

[0081] Data security: Invalid data may contain sensitive information or erroneous data, and transmitting it to the database may pose security risks; timely deletion of invalid data can reduce the risk of data leakage or data corruption and improve data security.

[0082] Based on the above scheme, the data acquisition system also includes a data processing module 107, which is used to perform synchronization processing on the data to be synchronized in the lock-free data buffer.

[0083] Secondly, this invention also provides a data acquisition method, see [link to relevant documentation]. Figure 2 The method includes:

[0084] Step 201: Collect the data required by the system;

[0085] Step 202: Temporarily store the collected system-required data in the data storage area of ​​the lock-free data buffer, and use the data management area in the buffer to classify and manage the data in the data storage area;

[0086] Step 203: Transfer the temporarily stored system data to the database using a multi-threaded concurrent execution method;

[0087] Step 204: Persistently store the data required by the system using the database and a nested linked list structure;

[0088] Step 205: Identify invalid data in the system's required data that does not meet the set requirements, and delete the invalid data.

[0089] Furthermore, in step 205 above, invalid data that does not meet the set requirements in the data required by the system is identified and deleted, including:

[0090] Monitor the system's required data temporarily stored in the lockless data buffer and identify invalid data that does not meet the set requirements;

[0091] Delete the identified invalid data stored in the database.

[0092] Furthermore, monitor the system's required data temporarily stored in the lock-free data buffer, and identify invalid data that does not meet the set requirements, including:

[0093] Monitor the data in the overflow buffer and invalid buffer within the lock-free data buffer;

[0094] The data in the overflow buffer and invalid buffer will be invalid data that does not meet the set requirements.

[0095] Furthermore, two sets of lock-free data buffers are provided, each set including a data storage area and a data management area;

[0096] Two sets of lock-free data buffers are used alternately to temporarily store and categorize the data required by the system.

[0097] Furthermore, the method also includes synchronizing the data to be synchronized in the lock-free data buffer.

[0098] The data acquisition method provided in this embodiment of the invention belongs to the same inventive concept as the aforementioned data acquisition system embodiment. Technical details not described in this embodiment can be found in the relevant descriptions in the aforementioned method embodiment, and will not be repeated here.

[0099] Although the invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, features in the embodiments disclosed herein can be combined in any way, provided there is no structural conflict. The lack of an exhaustive description of these combinations in this specification is merely for brevity and resource conservation. Therefore, the invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0101] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or substitutions made by a person skilled in the art that conform to the claims of the present invention regarding a cloud computing-based management system and management method should fall within the patent protection scope of the present invention.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0103] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0104] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0105] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0106] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0108] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A data acquisition system based on a lock-free buffer, characterized in that, include: The data acquisition module is used to collect the data required by the system and transmit it to the data buffer through the network module. The data buffer adopts a ring-linked data structure and is equipped with a lock-free data buffer. The data storage area within the lock-free data buffer is used to temporarily store data required by the system from the data acquisition module. The data management area of ​​the lock-free data buffer is used to classify and manage the data in the data storage area. Two sets of lock-free data buffers are created, each set including a data storage area and a data management area. The data buffer is used to alternately use the two sets of lock-free data buffers to temporarily store and classify the data required by the system. The transmission thread module is used to transmit the system's required data, which is temporarily stored in the data buffer, to the database in a multi-threaded concurrent execution manner. The database uses a nested linked list structure to persistently store the data transmitted by the transmission thread module. The monitoring module is used to identify invalid data in the system's required data that does not meet the set requirements, and to delete the invalid data.

2. The data acquisition system according to claim 1, characterized in that, The monitoring module is used to identify invalid data in the system's required data that does not meet the set requirements, and delete the invalid data, specifically including the following: Monitor the system's required data stored in the monitoring data buffer and identify invalid data that does not meet the set requirements; Delete the identified invalid data stored in the database.

3. The data acquisition system according to claim 2, characterized in that, The monitoring module is used to monitor the system's required data temporarily stored in the data buffer and identify invalid data that does not meet the set requirements, specifically including the following: Monitor the data in the overflow buffer and invalid buffer within the lock-free data buffer; Data in the overflow buffer and invalid buffer are treated as invalid data that does not meet the set requirements.

4. The data acquisition system according to claim 1, characterized in that, The system also includes a data processing module for synchronizing the data to be synchronized in the lock-free data buffer.

5. A data acquisition method based on a lock-free buffer, characterized in that, The method includes: Collect the data required by the system; The system temporarily stores the collected data in the data storage area of ​​the unlocked data buffer, and the data management area in the buffer is used to classify and manage the data in the data storage area. There are two sets of unlocked data buffers, each of which includes a data storage area and a data management area. The two sets of unlocked data buffers are used alternately to temporarily store and classify the data required by the system. The system uses a multi-threaded concurrent execution method to transfer temporarily stored system data to the database. The database uses a nested linked list structure to persistently store the data required by the system. Identify invalid data in the data required by the system that does not meet the set requirements, and delete the invalid data.

6. The data acquisition method according to claim 5, characterized in that, Identify and delete invalid data that does not meet the set requirements from the data required by the system, including: Monitor the system's required data temporarily stored in the lockless data buffer and identify invalid data that does not meet the set requirements; Delete the identified invalid data stored in the database.

7. The data acquisition method according to claim 6, characterized in that, Monitor the system's required data temporarily stored in the lock-free data buffer, and identify invalid data that does not meet the set requirements, including: Monitor the data in the overflow buffer and invalid buffer within the lock-free data buffer; Data in the overflow buffer and invalid buffer are treated as invalid data that does not meet the set requirements.

8. The data acquisition method according to claim 5, characterized in that, The system also includes a data processing module for synchronizing the data to be synchronized in the lock-free data buffer.

Images