A data efficient acquisition and storage method, system, device and storage medium

Abstract

This invention provides a method, system, device, and storage medium for efficient data acquisition and storage. The method includes: Step 1, the main thread initiates a data acquisition task; Step 2, the write thread is awakened to open a fixed-size circular data buffer; Step 3, the write thread executes a write operation to write the acquired data into the circular data buffer; Step 4, the read thread is awakened to execute a read operation based on the written data to obtain sub-stored data, and the sub-stored data is released from the address space corresponding to the circular data buffer; Step 5, Steps 3 and 4 are repeated to accumulate the amount of sub-stored data until the size of the sub-stored data reaches the size of the acquired file, thus obtaining the total stored data; Step 6, the total stored data is saved to a local file, waiting for the main thread to initiate the next data acquisition task, and then returning to Step 1. This method addresses packet loss issues, improves the efficiency and accuracy of data caching, and is suitable for the acquisition and storage of large amounts of data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data efficient collection and storage method, system device and storage medium. BACKGROUND

[0002] With the development of technology, people's processing tasks for data are getting heavier, and a large amount of valuable data often needs to be stored permanently in local high-speed data first, and then played back when needed. The requirement for data collection process is also getting higher and higher. When storing large capacity data, data loss often occurs due to transmission limitations and other problems, so that the completeness of the data cannot be guaranteed.

[0003] There are currently several technical solutions to solve the above problems: the first way is to open a larger memory in the memory, collect data into the memory, stop collecting when the application memory is used up, and then write the data to the hard disk; the second way is to use an interrupt to send the collected data block directly to the hard disk for storage; the third way is a double buffer mode, which writes data to the first buffer, moves to the second buffer for continuous writing when the first buffer is full, and uses multi-thread technology to read the first buffer. The first way is limited by memory space, and the storage data is limited, which cannot be stored for a long time; the second way is prone to packet loss, and a large amount of data is lost; the third way is prone to loss of individual data when changing the buffer. Therefore, it is necessary to study a high-performance data collection and storage method. SUMMARY

[0004] In order to solve the technical problems of low efficiency and poor accuracy of existing data collection and storage, the present application provides a data efficient collection and storage method, system, device and storage medium, which solves the problem of packet loss, improves the efficiency and accuracy of data caching, and can be applied to large data collection and storage.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a data efficient collection and storage method, which comprises the following steps,

[0007] Step 1: The main thread initiates a data collection task;

[0008] Step 2: Wake up the write thread to open a fixed-size ring-shaped data buffer;

[0009]

[0009] Step 3: The write thread executes a write operation process to write the collected data into the ring-shaped data buffer;

[0010] Step 4, the wake-up reading thread executes the reading operation procedure according to the data after the writing, obtains the sub-storage data, and releases the sub-storage data in the address space corresponding to the ring-shaped data buffer;

[0011] Step 5, steps 3 and 4 are repeatedly executed to continuously accumulate the amount of sub-storage data, until the size of the sub-storage data reaches the size of the collected file, and the total storage data is obtained;

[0012] Step 6, the total storage data is saved in the local file, and the next data collection task is initiated by the main thread, and returns to step 1.

[0013] Further, in step 3, the writing operation procedure includes the following steps:

[0014] Step 3.1, judge whether the remaining address space of the ring-shaped data buffer can store the collected data completely: if yes, execute step 3.2; if no, execute step 3.3;

[0015] Step 3.2, store the collected data to the ring-shaped data buffer, move the write pointer of the ring-shaped data buffer to the end of the current ring-shaped data buffer writing position, and calculate the current address information of the write pointer and the size of the remaining address space of the ring-shaped data buffer;

[0016] Step 3.3, store part of the collected data to the end of the ring-shaped data buffer, and store the un-stored part of the data from the head of the ring-shaped data buffer after the reading operation procedure releases the address space.

[0017] Further, in step 4, the reading operation procedure includes the following steps:

[0018] Step 4.1, judge whether the collected data has been stored to the minimum readable block: if yes, execute step 4.2; if no, execute step 4.3;

[0019] Step 4.2, move the read pointer of the ring-shaped data buffer to the end of the current ring-shaped data buffer reading position, and calculate the readable address space position, mark the read address space as a writable part;

[0020] Step 4.3, exit this reading, and wait for the next reading.

[0021] Further, the collection and storage method further includes step 7;

[0022] In step 1, the main thread needs to specify the size of the collected file and the position of the collected file before initiating the data collection task;

[0023] Step 7, specify the playback file, and use the playback software to play back the data stored in the local file.

[0024] In a second aspect, the present application provides a data efficient collection and storage system, which comprises:

[0025] a data collection initiation module for initiating a data collection task by a main thread;

[0026] a buffer area opening module for opening a fixed-size ring-shaped data buffer area by a write thread;

[0027] a data writing module for writing collected data into the ring-shaped data buffer area by the write thread;

[0028] a data reading module for reading data according to the written data by a read thread, obtaining sub-storage data, and releasing the sub-storage data in the corresponding address space of the ring-shaped data buffer area;

[0029] a data accumulation module for repeatedly executing the data writing module and the data reading module to continuously accumulate the amount of sub-storage data until the size of the sub-storage data reaches the size of the collected file, obtaining total storage data;

[0030] a data storage module for saving the total storage data in a local file and waiting for the main thread to initiate the next data collection task.

[0031] Further, the writing operation process in the data writing module comprises the following steps:

[0032] Step 3.1, judging whether the remaining address space of the ring-shaped data buffer area can completely store the collected data: if yes, executing Step 3.2; if no, executing Step 3.3;

[0033] Step 3.2, storing the collected data into the ring-shaped data buffer area, moving the write pointer of the ring-shaped data buffer area to the end of the current ring-shaped data buffer area write position, and calculating the current address information of the write pointer and the size of the remaining address space of the ring-shaped data buffer area;

[0034] Step 3.3, storing part of the collected data at the end of the ring-shaped data buffer area, and storing the remaining part of the data from the head of the ring-shaped data buffer area after the address space is released by the reading operation process.

[0035] Further, the reading operation process in the data reading module comprises the following steps:

[0036] Step 4.1, judging whether the collected data has been stored in the smallest readable block: if yes, executing Step 4.2; if no, executing Step 4.3;

[0037] Step 4.2, the read pointer of the annular data buffer is moved to the end of the annular data buffer read position, and the readable address space position is calculated, and the read address space is marked as a writable part;

[0038] Step 4.3, the current reading is exited, and the next reading is waited.

[0039] Further, the data collection initiation module needs to specify the collected file size and the collected file position before the main thread initiates the data collection task;

[0040] Further, the data collection initiation module needs to specify the collected file size and the collected file position before the main thread initiates the data collection task;

[0041] In a third aspect, the present application provides an electronic device, comprising:

[0042] A processor, a memory, and an interface for communicating with a gateway;

[0043] The memory is used for storing programs and data, and the processor calls the programs stored in the memory to execute the data efficient collection and storage method provided in any one of the first aspect.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, which comprises a program, and the program is used for executing the data efficient collection and storage method provided in any one of the first aspect when executed by a processor.

[0045] As can be seen from the above description, the embodiment of the present application provides a data efficient collection and storage method, which uses an annular data buffer to complete the high-speed collection and storage function of data, and only one address space is opened during the collection process, greatly reducing the operation of frequently opening addresses, effectively solving the problem of packet loss, and improving the accuracy of data caching; secondly, the sub-stored data needs to be accumulated to the collected file size before being stored in the local file, thereby reducing the storage frequency and improving the efficiency of data caching; and after obtaining the sub-stored data each time, the address space corresponding to the annular data buffer is released, so as to achieve the purpose of reusing the address space of the annular data buffer, get rid of the limitation of memory factors on large-capacity data collection and storage, and thus be suitable for large-capacity data collection and storage. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Fig. 1 shows a flowchart of the data efficient collection and storage method provided by an embodiment of the present application;

[0047] Figure 2 Fig. 2 shows a structural diagram of the data efficient collection and storage system provided by an embodiment of the present application;

[0048] Figure 3Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0050] Based on the disadvantages of the prior art, the present application provides a specific embodiment of a data efficient collection and storage method, as shown in Fig. 1, which specifically comprises the following steps: Figure 1

[0051] Step 1: initiating a data collection task by a main thread;

[0052] Specifically, in Step 1, the main thread needs to specify the collected file size and the collected file position before initiating the data collection task, so as to explicitly collect the task and reduce other redundant operations, thereby performing a purposeful data collection task and improving the efficiency of the data collection task.

[0053] Step 2: waking up a write thread to open a fixed-size circular data buffer;

[0054] Specifically, the size of the circular data buffer can be set to 512 KB, but it is not excluded that the size of the circular data buffer is set to other values according to actual conditions.

[0055] Step 3: the write thread performs a write operation process to write the collected data into the circular data buffer;

[0056] Specifically, in Step 3, the write operation process comprises the following steps:

[0057] Step 3.1: judging whether the remaining address space of the circular data buffer can completely store the collected data: if yes, performing Step 3.2; if no, performing Step 3.3;

[0058] Step 3.2: storing the collected data into the circular data buffer, moving the write pointer of the circular data buffer to the end of the current circular data buffer write position, and calculating the current address information of the write pointer and the size of the remaining address space of the circular data buffer;

[0059] Step 3.3: storing part of the collected data at the end of the circular data buffer, and storing the un-stored part of the data from the head of the circular data buffer in sequence after the address space is released by the read operation process.​

[0060] In step 3, the write operation flow is sequentially controlled by the write pointer, and the write pointer is moved downward according to the written data after each write operation flow is completed, so that the data written in the ring-shaped data buffer can be sequentially read by the read thread, and the data that cannot be written in the ring-shaped data buffer at one time is sequentially stored from the address space released by the read operation flow, forming a cycle of the write operation flow. The data write operation is sequentially cycled, which can effectively avoid the loss of collected data in the write process.

[0061] In step 4, the read thread according to the written data executes the read operation flow to obtain the sub-storage data, and releases the sub-storage data in the corresponding address space of the ring-shaped data buffer.

[0062] Specifically, in step 4, the read operation flow includes the following steps:

[0063] Step 4.1, determine whether the collected data has been stored in the smallest readable block: if yes, execute step 4.2; if no, execute step 4.3;

[0064] Step 4.2, move the read pointer of the ring-shaped data buffer to the end of the current ring-shaped data buffer reading position, and calculate the readable address space position, and mark the read address space as a writable part;

[0065] Step 4.3, exit this reading and wait for the next reading.

[0066] In step 4, before reading, it is necessary to determine whether the data collected in step 3 is stored in the smallest readable block that the read thread can read, which can effectively avoid resource waste caused by empty reading; in the data reading process, the read pointer is sequentially controlled, and after each read operation flow is completed, the read pointer is moved downward according to the data that has been read, and sub-storage data is generated when the address space of the ring-shaped data buffer is released each time, which ensures the integrity of data reading; the released address space can be used for the next write operation flow, achieving the purpose of repeated use of the ring-shaped data buffer.

[0067] Step 5, repeatedly execute steps 3 and 4 to continuously accumulate the amount of sub-storage data, until the size of the sub-storage data reaches the size of the collected file, to obtain the total storage data.

[0068] In this step, steps 3 and 4 are repeatedly executed, so that the sub-storage data needs to be accumulated to the size of the collected file before being stored in the local file, thereby reducing the storage times and improving the data caching efficiency.

[0069] Step 6, save the total storage data in the local file, wait for the main thread to initiate the next data collection task, and return to step 1;

[0070] Step 7, specify the playback file, and use the playback software to play back the data stored in the local file.

[0071] In step 7, the playback software is embedded in the lower machine in actual production, and the lower machine is usually connected with the signal device. By collecting and reading the data stored in the local file, the historical state of the signal device can be obtained, so as to realize the tracking of the state of the signal device. According to the device state message recorded by the lower machine, the historical data of the state of the signal device in the selected time period can be obtained. By gradually scanning the device state data every second, the historical signal device state and the troubleshooting of the signal device fault can be realized.

[0072] In summary, the application uses a ring-shaped data buffer area to complete the high-speed collection and storage function of data. Only one address space is opened during the collection process, which greatly reduces the operation of frequently opening addresses, effectively solves the problem of packet loss, and improves the accuracy of data caching. Secondly, the sub-storage data needs to be accumulated to the size of the collected file before being stored in the local file, thereby reducing the storage frequency and improving the efficiency of data caching. And after obtaining the sub-storage data each time, the address space corresponding to the ring-shaped data buffer area is released, so as to reuse the address space of the ring-shaped data buffer area, get rid of the limitation of memory factors in large-capacity data collection and storage, and then be suitable for large-capacity data collection and storage.

[0073] Based on the same inventive concept, the embodiments of the application also provide a data efficient collection and storage system, which can be used to realize the data efficient collection and storage method described in the above embodiments, as described in the following embodiments. Since the principle of solving problems of a data efficient collection and storage system is similar to that of a data efficient collection and storage method, the implementation of a data efficient collection and storage system can be referred to the implementation of a data efficient collection and storage method, and the repeated parts will not be described again. The term "unit" or "module" used below can be a combination of software and / or hardware that realizes a predetermined function. Although the system described in the following embodiments is preferably realized in software, the realization of hardware or a combination of software and hardware is also possible and conceived.

[0074] The application provides a data efficient collection and storage system, as shown in Figure 2 , in Figure 2 , the system comprises:

[0075] The data collection initiation module 210 is used for initiating a data collection task by a main thread.

[0076] The buffer opening module 220 is used to wake up the writing thread to open a fixed-size ring data buffer;

[0077] The data writing module 230 is used to write the collected data into the ring data buffer by the writing thread executing a writing operation process.

[0078] The data reading module 240 is used to obtain the sub-storage data by the reading thread executing a reading operation process according to the written data, and release the sub-storage data in the corresponding address space of the ring data buffer.

[0079] The data accumulation module 250 is used to repeatedly execute the processes of the data writing module and the data reading module to continuously accumulate the amount of sub-storage data until the size of the sub-storage data reaches the size of the collected file, and obtain total storage data.

[0080] The data storage module 260 is used to save the total storage data in a local file, and wait for the main thread to initiate the next data collection task.

[0081] Further, the writing operation process in the data writing module 230 of the data efficient collection and storage system according to an embodiment of the present application comprises the following steps:

[0082] Step 3.1, judging whether the remaining address space of the ring data buffer can completely store the collected data: if yes, executing step 3.2; if no, executing step 3.3.

[0083] Step 3.2, storing the collected data into the ring data buffer, moving the write pointer of the ring data buffer to the end of the current ring data buffer writing position, and calculating the current address information of the write pointer and the size of the remaining address space of the ring data buffer.

[0084] Step 3.3, storing part of the collected data into the end of the ring data buffer, and storing the un-stored part of the data from the head of the ring data buffer after the reading operation process releases the address space.

[0085] Further, the reading operation process in the data reading module 240 of the data efficient collection and storage system according to an embodiment of the present application comprises the following steps:

[0086] Step 4.1, judging whether the collected data has been stored into the minimum readable block: if yes, executing step 4.2; if no, executing step 4.3.

[0087] Step 4.2, moving the read pointer of the ring data buffer to the end of the current ring data buffer reading position, calculating the readable address space position, and marking the read address space as a writable part.

[0088] Step 4.3, jump out of this reading, and wait for the next reading.

[0089] Further, the data efficient collection and storage system of the embodiment of the present application, the main thread in the data collection initiation module 210 needs to specify the file size to be collected and the file location to be collected before initiating the data collection task;

[0090] Also includes a data playback module 270; for specifying the playback file, using the playback software to play back the data stored in the local file.

[0091] The embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps of the data efficient collection and storage method in the above-mentioned embodiment, see Figure 3 , the electronic device 300 specifically includes the following contents:

[0092] The processor 310, the memory 320, the communication unit 330 and the bus 340;

[0093] Among them, the processor 310, the memory 320, the communication unit 330 complete the communication between each other through the bus 340; the communication unit 330 is used to realize the information transmission between the server side device and the terminal device and other related devices.

[0094] The processor 310 is used to call the computer program in the memory 320, and the processor executes the computer program to realize all steps of the data efficient collection and storage method in the above-mentioned embodiment.

[0095] Those skilled in the art should understand that the memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) and the like. Among them, the memory is used to store programs, and the processor executes the programs after receiving the execution instructions. Further, the software programs and modules in the above-mentioned memory can also include an operating system, which can include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and can communicate with various hardware or software components, so as to provide running environment for other software components.

[0096] The processor can be an integrated circuit chip having a processing capability of signals. The processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc. The processor can implement or execute the disclosed methods, steps and logics in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc.

[0097] The present application also provides a computer readable storage medium, which comprises a program, the program being used for executing the data efficient collection and storage method provided by any one of the method embodiments when executed by a processor.

[0098] Those skilled in the art should understand that all or part of the steps of the above-mentioned data efficient collection and storage method embodiments can be completed by program instruction related hardware. The above-mentioned program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the above-mentioned storage medium includes ROM, RAM, magnetic disk or optical disk and various storage medium capable of storing program codes, and the specific type of the storage medium is not limited in the present application.

[0099] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for efficient data acquisition and storage, characterized by, The method comprises the following steps, Step 1, after specifying the collected file size and the collected file position, the main thread initiates a data collection task; Step 2, wake up the writing thread to open a fixed-size circular data buffer area; Step 3, the writing thread executes a writing operation process to write the collected data into the circular data buffer area; Step 4, wake up the reading thread to execute a reading operation process according to the written data, obtain sub-storage data, and release the sub-storage data in the corresponding address space of the circular data buffer area; Step 5, repeatedly execute steps 3 and 4 to continuously accumulate the amount of sub-storage data until the size of the sub-storage data reaches the collected file size, and obtain total storage data; Step 6, save the total storage data in a local file, wait for the main thread to initiate the next data collection task, and return to step 1.

2. The data efficient collection and storage method according to claim 1, wherein In step 3, the writing operation process comprises the following steps: Step 3.1, judge whether the remaining address space of the circular data buffer area can completely store the collected data: if yes, execute step 3.2; if no, execute step 3.3; Step 3.2, store the collected data into the circular data buffer area, move the write pointer of the circular data buffer area to the end of the current circular data buffer area writing position, and calculate the current address information of the write pointer and the size of the remaining address space of the circular data buffer area; Step 3.3, store part of the collected data at the end of the circular data buffer area, and wait for the reading operation process to release the address space to store the un-stored part of the data from the head of the circular data buffer area in sequence.

3. The data efficient collection and storage method according to claim 1, wherein In step 4, the reading operation process comprises the following steps: Step 4.1, judge whether the collected data has been stored in the smallest readable block: if yes, execute step 4.2; if no, execute step 4.3; Step 4.2, move the read pointer of the circular data buffer area to the end of the current circular data buffer area reading position, calculate the readable address space position, and mark the read address space as a writable part; Step 4.3, exit this reading, and wait for the next reading.

4. The method of claim 1, wherein, Further comprising step 7; In step 1, the main thread needs to specify the collected file size and the collected file position before initiating the data collection task; Step 7, specify a playback file, and use a playback software to play back the data stored in the local file.

5. A data efficient acquisition storage system, characterized by, Comprise: Data collection initiation module: used for the main thread to initiate a data collection task; Buffer area opening module: used for the writing thread to open a fixed-size circular data buffer area; Data writing module: used for the writing thread to execute a writing operation process to write the collected data into the circular data buffer area; Data reading module; used for the reading thread to execute a reading operation process according to the written data, obtain sub-storage data, and release the sub-storage data in the corresponding address space of the circular data buffer area; Data accumulation module; The flow for repeatedly executing the data writing module and the data reading module continuously accumulates the sub-storage data amount until the size of the sub-storage data reaches the size of the collected file, obtaining total storage data; The data storage module is used for saving the total storage data in a local file and waiting for the main thread to initiate the next data collection task.

6. The data efficient acquisition and storage system of claim 5, wherein, The writing operation flow in the data writing module includes the following steps: Step 3.1, judging whether the remaining address space of the ring-shaped data buffer can completely store the collected data: if yes, executing step 3.2; if no, executing step 3.3; Step 3.2, storing the collected data into the ring-shaped data buffer, moving the write pointer of the ring-shaped data buffer to the end of the current ring-shaped data buffer writing position, and calculating the current address information of the write pointer and the size of the remaining address space of the ring-shaped data buffer; Step 3.3, storing the collected data into the end of the ring-shaped data buffer, and storing the un-stored part of the data from the head of the ring-shaped data buffer in sequence after the address space is released by the reading operation flow.

7. The data efficient acquisition and storage system of claim 5, wherein, The reading operation flow in the data reading module includes the following steps: Step 4.1, judging whether the collected data has been stored into the minimum readable block: if yes, executing step 4.2; if no, executing step 4.3; Step 4.2, moving the read pointer of the ring-shaped data buffer to the end of the current ring-shaped data buffer reading position, calculating the readable address space position, and marking the read address space as a writable part; Step 4.3, exiting the current reading and waiting for the next reading.

8. The data efficient collection and storage system of claim 5, wherein Before the main thread initiates the data collection task in the data collection initiation module, the size of the collected file and the position of the collected file need to be specified; Further comprising a data playback module for specifying a playback file and playing back the data stored in the local file by using playback software.

9. An electronic device, comprising: It comprises: a processor, a memory, and an interface for communicating with a gateway; The memory is used for storing programs and data, and the processor invokes the programs stored in the memory to execute the data efficient collection and storage method of any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a program, which when executed by the processor, is used to execute the data efficient collection and storage method of any one of claims 1 to 4. The computer readable storage medium includes a program, which when executed by the processor, is used to execute the data efficient collection and storage method of any one of claims 1 to 4.

Images