A data storage and uploading method and apparatus for a data acquisition terminal

Abstract

This invention belongs to the technical field of data storage and communication, specifically relating to a data storage and uploading method and apparatus for a data acquisition terminal. The method includes the following steps: First, after the data acquisition terminal completes data acquisition, the acquired data is processed by an MCU main control unit and then uploaded to a cloud platform via a wireless communication module; second, an MD5 checksum algorithm is used to verify whether the data received by the platform is consistent with the data sent; if they are inconsistent, the data transmission fails; then, the failed data is written to an external Flash memory chip for storage; finally, the network status is checked periodically. When the network status is good and stored data exists, the data stored in the Flash memory chip is resent to the platform, and the storage area is cleared. This solves the problem of data loss due to network issues, provides a lightweight data storage solution for small outdoor embedded devices, reduces costs, and extends the lifespan of the Flash memory chip.

Description

Technical Field

[0001] This invention belongs to the technical field of data storage and communication, and more specifically, relates to a data storage and uploading method and apparatus for a data acquisition terminal. Background Technology

[0002] With the development of network technology, modern society is becoming increasingly information-based. The emergence of smart water conservancy, smart industry, and smart agriculture has further promoted the development of information technology, and all industries are undergoing digital transformation and upgrading. Currently, the main data collection methods are to collect data through sensors and then upload the data to the platform network through data collection terminals, or to directly collect data from sensors and upload it to the platform network. The latter has disadvantages such as high cost and limited data types, so data collection terminals are often needed to relay the data.

[0003] Chinese patent document CN114257978A discloses a method for transmitting sensor data from outdoor power transmission equipment. The method includes: acquiring the sensor data when the sensor sends sensor data to the relay node; the sensor data being the data collected by the sensor during monitoring of the outdoor power transmission equipment, and the sensor data representing the equipment status of the outdoor power transmission equipment; determining whether the relay node has received the sensor data; if it is determined that the relay node has not received the sensor data, then sending the sensor data to the smart gateway through the candidate node. This method can avoid retransmission due to excessive transmission distance when increasing the transmission distance per hop, effectively improving the transmission efficiency of sensor data. However, this method directly uses sensors to collect and upload data, which is very costly.

[0004] In urban areas with good network conditions, data acquisition terminals typically do not experience data transmission failures due to network issues. However, in practical applications, data acquisition terminals are often installed in vast fields or mountains where network signals are difficult to guarantee. Therefore, data loss due to poor network conditions frequently occurs. Currently, a common method is to store the data and then upload it. In embedded systems, due to storage resource limitations, external storage is often used for expansion. In small embedded devices, EEPROM or Flash memory is typically chosen as external storage. EEPROM has the advantages of long lifespan and can be erased byte-by-byte, but it is more expensive and has a relatively small memory capacity. Flash memory is relatively inexpensive and has a larger memory capacity, but it requires erasing in sector-by-sector units, resulting in a shorter lifespan. Existing data storage methods often suffer from high costs and complex system designs. Therefore, designing a more lightweight and cost-effective data storage and uploading method is crucial.

[0005] In view of this, the present invention designs a data storage and uploading method for a data acquisition terminal. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of the prior art and provide a data storage and uploading method for a data acquisition terminal, which effectively improves the data stability and efficiency of a simple embedded acquisition terminal, while also extending the lifespan of the Flash memory and reducing terminal costs.

[0007] The present invention also discloses an apparatus for data storage and uploading a data acquisition terminal.

[0008] The detailed technical solution of this invention is as follows:

[0009] A data storage and uploading method for a data acquisition terminal, the method comprising the following steps:

[0010] S1. After the data acquisition terminal completes the data acquisition, the acquired data is first controlled by the MCU main controller, and then uploaded to the cloud platform through the wireless communication module.

[0011] S2. Use the MD5 checksum algorithm to verify whether the data received by the cloud platform is consistent with the data sent. If they are inconsistent, the data transmission will fail.

[0012] S3. Write the failed data to an external Flash memory chip for storage;

[0013] The Flash memory chip determines memory allocation in the following way: a sector is divided into N parts, where N represents the number of data entries stored in this sector; M sectors are selected for data storage; if the Flash sector size is X, then the storage interval size is X / N; since Flash can only be erased in units of sectors, one sector is used as a transition area, and M sectors are used to store N*(M-1) data entries.

[0014] S4. A heartbeat mechanism is used to periodically detect the network status. When the network status is connected, the signal is good, and there is stored data, the data stored in the Flash storage chip is resent to the cloud platform, and the storage area is cleared.

[0015] According to a preferred embodiment of the present invention, step S2 specifically refers to the following steps: after the data is collected, the MCU main control performs MD5 verification and transmits the verification code and plaintext data. The platform side performs MD5 calculation again and compares the verification code to determine whether the data is complete. If the data is complete, the cloud platform returns a response code. Once the data acquisition terminal receives the response code, it means that the data transmission is complete. If the data acquisition terminal does not receive the response code or the MD5 verification code is inconsistent, the data transmission fails.

[0016] According to a preferred embodiment of the present invention, step S3, writing the failed data to an external Flash memory chip, includes the following steps:

[0017] S31. Determine if the number of currently stored data entries is greater than or equal to the set maximum value, i.e., N*(M-1): When the number of stored data entries is less than the maximum value, it proves that the number of stored data entries has not yet reached the maximum value, and there is still spare storage space. Then, increment the variable of the number of stored data entries by one and proceed to the judgment of the storage header address; when the number of stored data entries is greater than or equal to the maximum value, it means that the data has been stored up to the preset maximum number of entries, and then directly proceed to the judgment of the storage header address; since there is a transition area in the M sectors, when the stored data reaches the preset maximum value, the storage header address and storage tail address will automatically increase or decrease, and the two will not simultaneously satisfy the condition of being greater than or equal to the storage header address of the N-1th data entry in the Mth sector;

[0018] S32. Determine the storage head address and storage tail address: If both are less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, increment both the storage tail address and the storage head address by the storage interval size, and then proceed to the write address determination; if the storage head address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage head address to the storage head address, increment the storage tail address by the storage interval size, and then proceed to the write address determination; if the storage tail address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage tail address to the storage head address, increment the storage head address by the storage interval size, and then proceed to the write address determination.

[0019] S33. Since the transition area is used to receive data from one more sector to reduce the number of erase and write cycles of the Flash and extend its service life, it is necessary to determine the sector where the write address is located. When the number of stored data entries reaches the preset maximum value, if the write address is at the beginning address of a certain sector, then that sector is the transition area and can be directly erased before storing data.

[0020] Determine if the remainder of the write address divided by the sector capacity is 0: If it is 0, it indicates that the sector is a transitional area and can be erased before writing the data to be written, along with the number of stored data entries, the storage header address, and the storage tail address; if it is not 0, do not erase, and directly write the data to be written, along with the number of stored data entries, the storage header address, and the storage tail address.

[0021] According to a preferred embodiment of the present invention, in step S4, the specific steps for resending the data stored in the Flash memory chip to the cloud platform are as follows:

[0022] S41. Read the number of stored data entries, the storage header address, and the storage tail address into the system memory;

[0023] S42. Read data according to the number of stored data entries and send the read data: If the read data fails to be sent, determine whether the read data is not the first time it has failed to be sent and whether it has been sent successfully before. If not, enter the judgment area of ​​whether all data has been read and sent once; if so, write the current data to the space where the storage tail address is located, the storage tail address increments the storage interval size, and the storage tail address is written to the external storage, and then enter the judgment area of ​​whether all data has been read and sent once.

[0024] If the read data is successfully sent, determine if it is the first successful transmission. If so, decrement the number of stored data entries by one, change the storage tail address to the current read address, write the storage tail address and the number of stored data entries to external storage, and proceed to the current read address determination. If it is not the first successful transmission, decrement the number of stored data entries by one, write the number of stored data entries to external storage, and proceed to the current read address determination.

[0025] S43. Since the data is stored in a ring in the Flash memory, it is necessary to determine the current read address: if it is less than the storage start address of the (N-1)th data in the Mth sector, the current read address is incremented by the storage interval size, and then the system enters the judgment area to determine whether all data has been read and sent once; if it is not less than the storage start address of the (N-1)th data in the Mth sector, the current read address becomes the start address, and the system enters the judgment area to determine whether all data has been read and sent once.

[0026] S44. Since there may be multiple stored data entries, the stored data needs to be read out and sent sequentially: if not all data has been read out and sent once, the reading and sending process continues; if all data has been read out and sent once, it is determined whether all stored data has been sent. If all data has been sent, an empty data flag is written to external storage; if not all data has been sent, the current round of data reading and sending ends, and the unsuccessful data will be sent next time with priority.

[0027] In another aspect of the present invention, a data acquisition device is provided for implementing a data storage and uploading method for a data acquisition terminal, the device comprising:

[0028] An external Flash memory chip is used to store data that was not successfully sent.

[0029] The MCU master controller is responsible for the overall control and management of the entire system.

[0030] The wireless communication module is used to upload data to the cloud platform.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] In view of the problems of data packet loss of the data acquisition terminal, data transmission failure under poor network conditions, and short service life of external Flash storage, the present invention proposes a data storage and upload method for a data acquisition terminal. By using the MD5 verification algorithm to verify the integrity of each data transmission, the effective transmission of data is ensured; by adopting external Flash storage and setting a transition storage area, the effective storage of data with transmission failure is ensured, and the service life of Flash is extended; by periodically detecting the network conditions, it is ensured that the stored data can be timely sent to the platform. Brief Description of the Drawings

[0033] Figure 1 is the working flowchart of the data acquisition terminal of the present invention;

[0034] Figure 2 is the structural division diagram of the Flash storage chip of the present invention;

[0035] Figure 3 is the data storage flowchart of the present invention;

[0036] Figure 4 is the flowchart of reading and sending in-memory data of the present invention;

[0037] Figure 5 is the implementation effect diagram of the data acquisition terminal in the embodiment of the present invention. Detailed Embodiments <000008s>

[0038] The following further describes the present invention with reference to the drawings and embodiments.

[0039] Embodiment 1

[0040] Refer Figure 1 , a data storage and upload method for a data acquisition terminal, the method comprising the following steps:

[0041] S1. After the data acquisition terminal completes data acquisition, the acquired data is first controlled by the MCU main control, and then uploaded to the cloud platform through the wireless communication module;

[0042] S2. Use the MD5 verification algorithm to check whether the data received by the cloud platform is consistent with the sent data. If not, the data transmission fails;

[0043] S3. Write the data with transmission failure into the external Flash storage chip for storage; <000009s>S4. Adopt a heartbeat mechanism to periodically detect the network connection status. When the network connection status is connected, the signal is good and there is stored data, the data stored in the Flash storage chip is resent to the cloud platform, and the storage area is cleared.

[0045] Step S2 specifically refers to using the MD5 checksum algorithm to check data integrity. After data acquisition, the MCU main control performs MD5 verification and transmits the checksum and plaintext data. The platform side determines whether the data is complete by performing MD5 calculation again and comparing the checksum. If the data is complete, the platform returns a response code. Once the data acquisition terminal receives the response code, it means that the data transmission is complete. If the data acquisition terminal does not receive a response code, or the MD5 checksum is inconsistent, the data transmission is considered to have failed.

[0046] like Figure 2 As shown, the Flash memory chip determines memory allocation as follows: a sector is divided into N parts, where N represents the number of data entries stored in this sector; M sectors are selected for data storage. If the Flash sector size is X, then the storage interval size is X / N. Since Flash can only be erased in sector units, one sector is used as a transition area, and M sectors are used to store N*(M-1) data entries.

[0047] like Figure 3 As shown, step S3, writing the failed data to the external Flash memory chip, includes the following steps:

[0048] S31. First, determine whether the number of currently stored data entries is greater than or equal to the set maximum value, i.e., N*(M-1). If the number of stored data entries is less than N*(M-1), it proves that the number of stored data entries has not reached the maximum value and there is still spare storage space. Then, increment the variable of the number of stored data entries by one and enter the judgment of the storage header address. If the number of stored data entries is greater than or equal to the maximum value, it means that the data has been stored up to the preset maximum number of entries, and then directly enter the judgment of the storage header address. Since there is a transition area in the M sectors, when the stored data reaches the preset maximum value, the storage header address and storage tail address will automatically increase or decrease, and the two will not simultaneously satisfy the condition of being greater than or equal to the storage header address of the N-1th data entry in the Mth sector.

[0049] S32. Determine the storage head address and storage tail address. If both are less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, increment both the storage tail address and the storage head address by the storage interval size, and then proceed to the write address determination. If the storage head address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage head address to the storage head address, increment the storage tail address by the storage interval size, and then proceed to the write address determination. If the storage tail address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage tail address to the storage head address, increment the storage head address by the storage interval size, and then proceed to the write address determination.

[0050] S33. Since this solution uses a transition area to receive data from an additional sector in order to reduce the number of times the Flash is erased and rewritten, thus extending its service life, it is necessary to determine the sector where the write address is located. When the number of stored data entries reaches the preset maximum value, if the write address is at the beginning address of a certain sector, then that sector is the transition area, which can be directly erased before storing data.

[0051] Therefore, it is necessary to determine whether the remainder of the write address divided by the sector capacity is 0. If it is, it indicates that the sector is a transitional area and can be erased. Then, the data to be written is written, along with the number of stored data entries, the storage header address, and the storage tail address. If it is not, the sector is not erased, and the data to be written is written directly, along with the number of stored data entries, the storage header address, and the storage tail address.

[0052] Figure 3 The number of stored data entries in the value indicates the number of data entries currently stored in the Flash memory.

[0053] The maximum value represents the number of data entries that can be stored in the Flash memory, which is N*(M-1) in this case.

[0054] This invention employs a ring storage method, where the storage header address represents the starting address of the storage space containing the earliest piece of data among all currently stored data.

[0055] The storage tail address represents the starting address of the storage space where the last piece of data is located among all the data currently stored;

[0056] The tail data address represents the starting address of the storage space allocated in Flash to store the last piece of data that has not yet been sent. Here, it is the storage starting address of the (N-1)th piece of data in the Mth sector.

[0057] The write address refers to the starting address of the storage space where the data is to be written;

[0058] The starting address represents the starting address of all the space allocated in Flash to store the failed data transmission.

[0059] Sector capacity refers to the number of bytes that a single sector of Flash memory can store; here, it is represented by X.

[0060] The storage interval size represents the space occupied by one piece of data in Flash, which is X / N in this case.

[0061] like Figure 4 As shown, in step S4, the data stored in the Flash memory chip is resent to the platform, which needs to read and send the stored data. The specific steps are as follows:

[0062] S41. Read the number of stored data entries, the storage header address, and the storage tail address into the system memory;

[0063] S42. Read data according to the number of stored data entries, and send the read data;

[0064] S43. If the data fails to be sent, determine whether it is not the first time it has failed to be sent and whether it has been sent successfully before. If not, enter the judgment area of ​​whether all data has been read and sent once. If yes, write the current data into the space where the storage tail address is located, increment the storage tail address by the storage interval size, write the storage tail address into the external storage, and then enter the judgment area of ​​whether all data has been read and sent once.

[0065] S44. If the data has been successfully read and sent, determine whether it is the first successful transmission. If so, decrement the number of stored data entries by one, change the storage tail address to the current read address, write the storage tail address and the number of stored data entries to the external storage, and proceed to the current read address determination. If it is not the first successful transmission, decrement the number of stored data entries by one, write the number of stored data entries to the external storage, and proceed to the current read address determination.

[0066] S45. Since the data is stored in a ring in the Flash, it is necessary to judge the current read address. If it is less than the storage start address of the N-1th data in the Mth sector, the current read address is incremented by the storage interval size, and then the judgment area for whether all data has been read and sent once is entered. If it is not less than the storage start address of the N-1th data in the Mth sector, the current read address becomes the start address, and the judgment area for whether all data has been read and sent once is entered.

[0067] S46. Since there may be multiple stored data entries, the stored data needs to be read out and sent sequentially. If not all data has been read out and sent once, the reading and sending process continues. If all data has been read out and sent once, it is determined whether all stored data has been sent. If all data has been sent, an empty data flag is written to external storage. If not all data has been sent, the current round of data reading and sending ends. Data that was not successfully sent will be sent in the next round with priority.

[0068] Example 2

[0069] This embodiment provides a data acquisition device for implementing a data storage and uploading method for a data acquisition terminal. The device includes:

[0070] An external Flash memory chip is used to store data that was not successfully sent.

[0071] The MCU master controller is responsible for the overall control and management of the entire system.

[0072] The wireless communication module is used to upload data to the cloud platform.

[0073] This data acquisition terminal is used for data collection from agricultural sensors in mountainous areas. Agricultural sensors generally support RS485 protocol transmission; therefore, this data acquisition terminal uses RS485 input, processes the data via the MCU main control unit, packages it, and uploads it to the monitoring cloud platform via a 4G module. However, signal strength is often weak in mountainous areas, especially during rainy weather, leading to frequent data transmission failures from agricultural sensors. Therefore, the data acquisition terminal proposed in this invention can promptly transmit data when the signal is good, and save the agricultural sensor data when the signal is poor. Once the signal is restored, the data for that time period is uploaded, thus preventing discontinuities in the agricultural sensor data over time. Its implementation effect is as follows: Figure 5 As shown:

[0074] The components are: 1: 4G antenna, 2: pole, 3: waterproof box, 4: data acquisition terminal, 5: power cord, 6: agricultural sensor, and 7: RS485 data cable.

Claims

1. A data storage and uploading method for a data acquisition terminal, characterized in that, The method includes the following steps: S1. After the data acquisition terminal completes the data acquisition, the acquired data is first controlled by the MCU main controller, and then uploaded to the cloud platform through the wireless communication module. S2. Use the MD5 checksum algorithm to verify whether the data received by the cloud platform is consistent with the data sent. If they are inconsistent, the data transmission will fail. S3. Write the failed data to an external Flash memory chip for storage; The Flash memory chip determines memory allocation in the following way: a sector is divided into N parts, where N represents the number of data entries stored in this sector; M sectors are selected for data storage; if the Flash sector size is X, then the storage interval size is X / N; a sector is used as a transition area, and M sectors are used to store N*(M-1) data entries. The process of writing failed data to an external Flash memory chip specifically includes the following steps: S31. Determine if the number of currently stored data entries is greater than or equal to the set maximum value, i.e., N*(M-1): When the number of stored data entries is less than the maximum value, increment the variable of the number of stored data entries by one and proceed to the judgment of the storage header address; when the number of stored data entries is greater than or equal to the maximum value, proceed to the judgment of the storage header address; when the number of stored data entries reaches the preset maximum value, the storage header address and storage tail address will be automatically increased or decreased, and neither of them will simultaneously satisfy the condition of being greater than or equal to the storage header address of the N-1th data entry in the Mth sector; S32. Determine the storage head address and storage tail address: If both are less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, increment both the storage tail address and the storage head address by the storage interval size, and then proceed to the write address determination; if the storage head address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage head address to the storage head address, increment the storage tail address by the storage interval size, and then proceed to the write address determination; if the storage tail address is not less than the storage head address of the (N-1)th data in the Mth sector, change the write address to the storage tail address, change the storage tail address to the storage head address, increment the storage head address by the storage interval size, and then proceed to the write address determination. S33. Determine if the remainder of the write address divided by the sector capacity is 0: If it is 0, it indicates that the sector is a transition area and can be erased. Then, the data to be written is written, along with the number of stored data entries, the storage header address, and the storage tail address. If it is not 0, the sector is not erased, and the data to be written is written directly, along with the number of stored data entries, the storage header address, and the storage tail address. S4. A heartbeat mechanism is used to periodically detect the network status. When the network status is connected, the signal is good, and there is stored data, the data stored in the Flash storage chip is resent to the cloud platform, and the storage area is cleared.

2. The data storage and uploading method of a data acquisition terminal according to claim 1, characterized in that, Step S2 specifically refers to the following steps: After the data is collected, the MCU main control performs MD5 verification and transmits the verification code and plaintext data. The platform side performs MD5 calculation again and compares the verification code to determine whether the data is complete. If the data is complete, the cloud platform returns a response code. Once the data acquisition terminal receives the response code, it means that the data transmission is complete. If the data acquisition terminal does not receive the response code or the MD5 verification code is inconsistent, the data transmission fails.

3. The data storage and uploading method of a data acquisition terminal according to claim 2, characterized in that, In step S4, the specific steps for resending the data stored in the Flash memory chip to the cloud platform are as follows: S41. Read the number of stored data entries, the storage header address, and the storage tail address into the system memory; S42. Read data according to the number of stored data entries and send the read data: If the read data fails to be sent, determine whether the read data is not the first time it has failed to be sent and whether it has been sent successfully before. If not, enter the judgment area of ​​whether all data has been read and sent once; if so, write the current data to the space where the storage tail address is located, the storage tail address increments the storage interval size, and the storage tail address is written to the external storage, and then enter the judgment area of ​​whether all data has been read and sent once. If the read data is successfully sent, then determine whether it is the first successful transmission. If so, decrement the number of stored data entries by one, change the storage tail address to the current read address, write the storage tail address and the number of stored data entries to external storage, and enter the current read address judgment process. If the first transmission is not successful, decrement the number of stored data entries by one, write the number of stored data entries to external storage, and proceed to the current read address determination. S43. Determine the current read address: If it is less than the storage start address of the N-1th data in the Mth sector, then increment the current read address by the storage interval size, and then enter the judgment area to determine whether all data has been read and sent once. If the data is not less than the storage starting address of the (N-1)th data in the Mth sector, then the current read address becomes the starting address, and the system enters the judgment area to determine whether all data has been read and sent once. S44. Read out and send the stored data sequentially: If not all data has been read out and sent once, continue reading and sending; if all data has been read out and sent once, determine whether all stored data has been sent. If all data has been sent, write an empty data flag to external storage; if not all data has been sent, the current round of data reading and sending ends, and the unsuccessful data will be sent next time with priority.

4. An apparatus for implementing the data storage and uploading method of the data acquisition terminal as described in any one of claims 1-3, characterized in that, The device includes: An external Flash memory chip is used to store data that was not successfully sent. The MCU master controller is responsible for the overall control and management of the entire system. The wireless communication module is used to upload data to the cloud platform.

Images