Processing method for coping with millisecond-level mass data gushing of serial port

Abstract

The invention belongs to the field of power systems, and provides a processing method for coping with millisecond-level mass data gushing of a serial port. Comprising the following steps: S1, determining the specification of a serial port, a serial port line and a serial port line chip; s2, the serial port receives the original data and stores the original data in the cache region, and the stack region obtains the data from the cache region in real time; s3, regularly extracting data of a preset duration from the stack area, storing the data into a relational database, and adding a timestamp; s4, taking out the data according to a time sequence, setting a buffer processing area, and splicing byte arrays; s5, carrying out CRC (Cyclic Redundancy Check); s6, comparing data types of adjacent byte arrays; and S7, obtaining the byte array after comparison in the step S6, analyzing the byte array into plaintext data, and storing the plaintext data into a database. According to the invention, data integrity can be ensured, and data processing efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the field of power systems, and in particular relates to a processing method for coping with the millisecond-level massive data outflow of a serial port. Background Art

[0002] In the power system, the RS485 network is used to realize the communication between the energy meter and the meter reading concentrator. The meter reading concentrator collects the data of multiple energy meters through the RS485 network and uploads it to the server of the power management department, realizing remote automatic meter reading, improving the efficiency and accuracy of meter reading, and reducing labor costs. In the field of industrial data communication, the baud rate supported by the serial port of the RS485 network and related equipment is set to 115200 in normal operation according to standard specifications. However, with the continuous increase in the monitoring data business volume during the operation of the system, especially when the amount of data to be received within 1 second reaches 25.6KB, the transmission capacity of the preset baud rate can no longer fully meet the actual needs of data transmission. This mismatch causes the serial port to be blocked, and the data transmission efficiency of the serial port then shows a rapid decline. This makes it inevitable that a large number of bytes will be lost during the communication process, which will have a serious impact on the accuracy, integrity of the entire data communication, and the stability and reliability of the system. It may cause the subsequent data processing, analysis, and related control operations that rely on the data communication link to fail to be carried out normally and effectively, and may even cause a series of chain problems such as partial functional failure or serious performance degradation of the entire monitoring and control system. In addition, since individual threads involve database persistence operations, the operation may have delay problems. Therefore, in places where delays may occur, caches should be used to solve the delay problem, and threads should not affect or block each other to improve the overall efficiency of software processing.

[0003] Patent document CN201810286486.8 discloses a reliable polling communication method for power line carrier, which uses protocol frames as carriers and includes the following key steps: designing the network layer protocol format (including host address, slave address, etc.), first verifying the data, then judging the frame sequence number and protocol frame status bit in turn, and letting the slave judge whether the protocol frame last sent to the host has data according to the status bit (when it is 0), as well as timeout processing steps. This method can reduce the packet loss rate of the communication system, realize multi-point high-speed reliable polling communication on power lines in industrial harsh channels, have good QoS guarantee, meet the high reliability requirements of information collection and reduce the network failure rate.

[0004] However, the existing technology has at least the following technical problems: The existing technology has not conducted research on RS485 network as the application object, so there are differences in application implementation. The existing technology only solves the problem of byte loss from the aspects of protocol and data frame verification, and does not systematically optimize the communication failure caused by large-capacity data rapid transmission from the aspects of transmission media, data receiving method, and data verification method, resulting in the inability to complete accurate transmission when the communication business volume suddenly increases. Summary of the invention

[0005] The present invention aims to solve the technical problems existing in the prior art and provide a processing method for coping with the millisecond-level large-scale data outflow from a serial port. The method can ensure the integrity of the data received by the serial port, and build an efficient data temporary storage mechanism through caching technology. At the moment the serial port receives data, the data can be quickly placed in the designed cache area, avoiding the dead time caused by waiting for the serial port to transmit byte by byte, thereby improving the system data processing efficiency.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: A method for processing a large amount of data flowing out of a serial port at the millisecond level, comprising the following steps: S1. Debug the serial port, switch the serial port baud rate required for millisecond-level data transmission, and then determine the serial port specifications, and determine the serial port line and serial port line chip according to the serial port specifications; S2, the serial port receives the original data and stores it in the buffer area, the stack area obtains data from the buffer area in real time, and deletes the obtained data at the same time; S3, start the cyclic extraction mode, regularly extract the data of the preset time length from the stack area, serialize and store it in the queue, and extract and store the data in the queue into the relational database, and add a timestamp; S4, traversing and taking out the data in the relational database in time order, converting the data into byte data, and setting a buffer processing area, and performing byte array splicing on the byte data in the buffer processing area; S5, verifying the byte data in the byte array through CRC checksum, and determining whether the data type of the current byte array is voltage or current; S6, obtaining the byte array and data type that pass the CRC checksum, comparing the data types of adjacent byte arrays in chronological order, and if they are both voltage or current, discarding the byte array arranged at the back, until the data types of the byte arrays are different; S7. Obtain the byte array compared in step S6, parse it into plain text data, and store it in a database.

[0007] Optionally, in step S1, the inductive reactance of the serial communication line is calculated using the baud rate configuration derivation method, and then the serial port specification is determined. The calculation process of the inductive reactance of the serial communication line is as follows: Calculate the inductance of the circular cross-section straight conductor of the serial communication line. The calculation formula is as follows: , in, is the inductance of the circular cross-section straight conductor of the serial communication line, is the vacuum permeability, is the wire length, is the wire radius; Calculate the inductive reactance of the serial communication line, the formula is as follows: , in, is the inductive reactance of the serial communication line, is the frequency.

[0008] Optionally, in step S2, the original data is sent to the selected serial port, the original data is in the form of a two-dimensional array, the serial port includes a buffer area and a stack area, the serial port receives data and stores the received data in the buffer area, the stack area obtains data from the buffer area in real time and deletes the obtained data at the same time.

[0009] Optionally, in step S3, the data of the preset duration is serialized by a JSON serialization method, the data of the preset duration is converted into a string, and stored in a queue, and the data of the preset duration that has been extracted from the stack area is deleted.

[0010] Optionally, in step S3, when the cyclic extraction mode is started, the data in the queue is extracted according to a preset time length through a preset sub-thread, and stored in a relational database, and a timestamp corresponding to the stored data is added.

[0011] Optionally, in step S4, after traversing and retrieving the data in the relational database in chronological order, the data in the relational database is first converted into a two-dimensional data structure in bytes through JSON deserialization. A buffer processing area is defined for different byte conditions of the converted data. In the buffer processing area, a preset a bytes are used as a byte array, and the converted data is continued to be traversed. The part exceeding a bytes or less than a bytes is put into the buffer processing area and spliced ​​to the front of the next data to form a byte array.

[0012] Optionally, in step S5, 5 bytes are used as a byte group number, and the converted data is grouped in the form of index positioning, and each group of data is CRC-checked and verified according to 5 bytes.

[0013] Optionally, in step S5, for the byte array that fails the check, the byte array is reassembled by sliding one byte to continue calculating the CRC checksum until the checksum passes.

[0014] Optionally, in step S6, when comparing the data types of adjacent byte arrays in chronological order, if they are both voltage or current, it is determined that data is lost, the byte array arranged at the rear is discarded, and the next byte array is added until the data types of adjacent byte arrays are different.

[0015] Optionally, in step S7, the compared byte array is parsed into plaintext data, and the plaintext data is placed in a two-dimensional array in the order of parsing, consistent with the form of the original data. When the plaintext data in the two-dimensional array is complete original data, the plaintext data is matched with the timestamp added in step S3 and stored in the database.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) The present invention realizes the combined use of multithreading, cache technology, and stack buffer technology by presetting sub-threads, thereby ensuring the integrity of the data received by the serial port, avoiding serial port blocking, and thus shortening the time for subsequent data analysis and storage; (2) In the dynamic process of the serial port of the present invention continuously receiving a large amount of data, the preset sub-thread can be responsible for close interaction with the port, accurately and in real time capturing the continuously influx of data streams, so that it is not interfered by other software-level operation delays, minimizing the risk of data omissions and firmly ensuring the integrity of the data receiving link; (3) The present invention constructs an efficient data temporary storage mechanism through the cache technology of the serial port buffer area and the stack area. At the moment the serial port receives data, the data can be quickly placed in the designed stack area. When entering the data parsing stage later, the multi-threaded architecture allows independent parsing threads to quickly retrieve data directly from the stack area, avoiding the stagnation time caused by waiting for the serial port to transmit byte by byte, greatly speeding up the parsing process, and comprehensively improving the data processing efficiency and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a method for processing a large amount of millisecond-level data outflow from a serial port according to an embodiment of the present invention; Figure 2 The present invention provides a technical roadmap for a method for processing a large amount of data flowing out of a serial port at the millisecond level according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Example 1 Combination Figure 1-Figure 2 As shown, the present invention provides a method for processing a large amount of data flowing out of a serial port at the millisecond level, comprising the following steps: S1. Debug the serial port, switch the serial port baud rate required for millisecond-level data transmission, and then determine the serial port specifications, and determine the serial port line and serial port line chip according to the serial port specifications; S2, the serial port receives the original data and stores it in the buffer area, the stack area obtains data from the buffer area in real time, and deletes the obtained data at the same time; S3, start the cyclic extraction mode, regularly extract the data of the preset time length from the stack area, serialize and store it in the queue, and extract and store the data in the queue into the relational database, and add a timestamp; S4, traversing and taking out the data in the relational database in time order, converting the data into byte data, and setting a buffer processing area, and performing byte array splicing on the byte data in the buffer processing area; S5, verifying the byte data in the byte array through CRC checksum, and determining whether the data type of the current byte array is voltage or current; S6, obtaining the byte array and data type that pass the CRC checksum, comparing the data types of adjacent byte arrays in chronological order, and if they are both voltage or current, discarding the byte array arranged at the back, until the data types of the byte arrays are different; S7. Obtain the byte array compared in step S6, parse it into plain text data, and store it in a database.

[0020] Example 2 Compared with Example 1, in this embodiment, in step S1, since the inductance of the serial communication line will affect the signal transmission capability, signal consumption and transmission distance, it is necessary to evaluate and debug the serial port according to the inductance of the serial communication line to determine the serial port specifications.

[0021] like Figure 2 As shown in the technical roadmap, the serial port specifications are determined, the serial port is debugged, and the baud rate is switched as follows: Assuming that the initial specification of the serial port (initial baud rate) is 19200bps, when the serial port specification does not meet the demand for large amounts of data to flow out at the millisecond level, the serial port is debugged. At this time, the serial port specification can be switched to a serial port with a baud rate of 1.5Mbps to meet the communication needs for large amounts of data to flow out at the millisecond level.

[0022] Regarding the determination of the serial port baud rate, the baud rate configuration derivation method can be used to calculate the inductive reactance of the serial port communication line, and then determine the serial port baud rate, and further determine the serial port specifications. The calculation process of the inductive reactance of the serial port communication line is as follows: Calculate the inductance of the circular cross-section straight conductor of the serial communication line. The calculation formula is as follows: , in, is the inductance of the circular cross-section straight conductor of the serial communication line, is the vacuum permeability, is the wire length, is the wire radius; Calculate the inductive reactance of the serial communication line, the formula is as follows: , in, is the inductive reactance of the serial communication line, is the frequency; That is to say, the frequency The larger it is, the greater the inductive reactance is, and the greater the ability to block the signal is.

[0023] For the RS485 communication used by the chip to transmit 26.4KB of data per second, after multiple serial port evaluations and debugging, it was finally determined to use a baud rate of 1.5Mbps for communication. The BL1590 high-speed chip was selected, and the serial port line of the BL1590 chip with relatively stable performance was used as the communication line.

[0024] Furthermore, in step S2, the original data is sent to the selected serial port, the original data is in the form of a two-dimensional array, the serial port includes a buffer area and a stack area, the serial port receives data and stores the received data in the buffer area, the stack area obtains data from the buffer area in real time, and deletes the obtained data at the same time.

[0025] Further, in step S3, the data of the preset duration is serialized by a JSON serialization method, the data of the preset duration is converted into a string, and stored in a queue, and the data of the preset duration that has been extracted from the stack area is deleted; Furthermore, when the cyclic extraction mode is turned on, the data in the queue is extracted according to the preset time length through the preset sub-thread, and stored in the relational database, and a timestamp corresponding to the stored data is added.

[0026] Example 3 Compared with Example 1, in this embodiment, in step S4, after traversing and taking out the data in the relational database in chronological order, the data in the relational database is first converted into a two-dimensional data structure in bytes through JSON deserialization, and a buffer processing area is defined for different byte conditions of the converted data. In the buffer processing area, a preset a bytes are used as a byte array, and the converted data is continued to be traversed. The part exceeding a bytes or less than a bytes is put into the buffer processing area and spliced ​​to the front of the next data to form a byte array.

[0027] As a preferred manner, in step S5, the converted data is grouped in the form of index positioning with 5 bytes as a byte group number, and each group of data is subjected to CRC checksum check according to 5 bytes, wherein the CRC checksum is a cyclic redundancy check (Cyclic Redundancy Check).

[0028] Furthermore, in step S5, for the byte array that fails the check, the byte array is reassembled by sliding one byte to continue calculating the CRC checksum until the checksum passes.

[0029] Example 4 Compared with Example 1, in this embodiment, after the serial port receives data, the data is stored in the serial port buffer area, and the stack area takes data from the buffer area in real time, which will not block the serial port from receiving data, so that the data received by the serial port can be saved in the stack area; after the data is taken out, the data taken out from the buffer area is deleted to ensure that the data in the buffer area will not overflow, thereby ensuring continuous reception of the serial port; Step S3 is specifically as follows: start the loop extraction mode, regularly extract the data of the preset time length from the stack area, serialize the data of the preset time length through the JSON serialization method, convert the data of the preset time length into a string, and store it in the queue, delete the data of the preset time length that has been extracted from the stack area, and extract the data in the queue according to the preset time length through a preset sub-thread, store it in the relational database, and add a timestamp; When the loop extraction mode is turned on, 3 seconds of data are taken out from the stack area in a loop, and the taken out 3 seconds of data are JSON serialized into the queue, and then the data within 3 seconds taken out from the stack area is deleted to ensure that the data in the stack area will not overflow. This method can ensure that the data saving speed is greater than 3 seconds, that is, greater than the speed of receiving data at the serial port; The stack area solves the problem of data loss when the amount of data is large and the data storage takes a long time, ensuring the integrity of data storage.

[0030] Step S4 specifically includes: traversing and extracting the data in the relational database in chronological order, and converting the data in the relational database into a two-dimensional data structure in bytes through JSON deserialization. This is because the extracted original data is currently in the form of a two-dimensional array of bytes, so it needs to be converted into a string through JSON deserialization; The original data is stored in the relational database in the form of a byte two-dimensional array (Note: C# original two-dimensional array byte[][], in the real code, the more efficient linear table storage provided by .net will be used); In view of the different byte conditions of the converted data, a buffer processing area is defined. In the buffer processing area, a preset a byte is used as a byte array. The converted data is continuously traversed. For the part exceeding a bytes or less than a bytes, it is put into the buffer processing area and spliced ​​to the front of the next data to form a byte array. Step S5 is specifically as follows: taking 5 bytes as a byte group number, the converted data is grouped in the form of index positioning, and each group of data is subjected to CRC checksum verification according to 5 bytes. For the byte array that fails the checksum, the byte array is reassembled by sliding one byte to continue calculating the CRC checksum until the checksum passes; At the same time, according to the difference in CRC checksum format between voltage data and current data, it is determined whether the data type of the current byte array is voltage or current; Step S6 is specifically as follows: obtaining the byte array and data type that pass the CRC checksum, and comparing the types of adjacent byte arrays in chronological order. If they are both voltage or current, it is determined that data is lost, and the byte array arranged at the back is discarded, and the next byte array is filled in until the types of adjacent byte data are different; Among them, because current and voltage appear in pairs, each time the data is calculated, it is necessary to compare it with the previous data. If the two data are both voltage or current, it means that there is data loss in the middle. At this time, the data should be discarded and the next data calculation should be continued until the data that is different from the last data belongs to the same category. Step S7 is specifically as follows: obtaining the byte array after comparison in step S6, parsing the compared byte array into plain text data, and placing the plain text data into a two-dimensional array in the order of parsing, so as to keep it consistent with the form of the original data; when the plain text data in the two-dimensional array is the complete original data, matching the plain text data with the timestamp added in step S3 and storing it in the database.

[0031] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the application scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing a large amount of data flowing out of a serial port at the millisecond level, characterized in that: The following steps are involved: S1. Debug the serial port, switch the serial port baud rate required for millisecond-level data transmission, and then determine the serial port specifications, and determine the serial port line and serial port line chip according to the serial port specifications; S2, the serial port receives the original data and stores it in the buffer area, the stack area obtains data from the buffer area in real time, and deletes the obtained data at the same time; S3. Start the cyclic extraction mode, regularly extract the data of the preset time length from the stack area, serialize and store it in the queue, and extract and store the data in the queue into the relational database, and add a timestamp; S4, traversing and taking out the data in the relational database in time order, converting the data into byte data, and setting a buffer processing area, and performing byte array splicing on the byte data in the buffer processing area; S5, verifying the byte data in the byte array through CRC checksum, and determining whether the data type of the current byte array is voltage or current; S6, obtaining the byte array and data type that pass the CRC checksum, comparing the data types of adjacent byte arrays in chronological order, and if they are both voltage or current, discarding the byte array arranged at the back, until the data types of the byte arrays are different; S7. Obtain the byte array compared in step S6, parse it into plain text data, and store it in a database.

2. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S1, the inductive reactance of the serial communication line is calculated using the baud rate configuration derivation method, and then the serial port specifications are determined. The calculation process of the inductive reactance of the serial communication line is as follows: Calculate the inductance of the circular cross-section straight conductor of the serial communication line. The calculation formula is as follows: ， in, is the inductance of the circular cross-section straight conductor of the serial communication line, is the vacuum permeability, is the wire length, is the wire radius; Calculate the inductive reactance of the serial communication line, the formula is as follows: ， in, is the inductive reactance of the serial communication line, is the frequency.

3. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S2, the original data is sent to the selected serial port. The original data is in the form of a two-dimensional array. The serial port includes a buffer area and a stack area. The serial port receives data and stores the received data in the buffer area. The stack area obtains data from the buffer area in real time and deletes the obtained data at the same time.

4. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S3, the data of the preset duration is serialized by the JSON serialization method, the data of the preset duration is converted into a string, and stored in the queue, and the data of the preset duration that has been extracted from the stack area is deleted.

5. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 4, characterized in that: In step S3, when the cyclic extraction mode is started, the data in the queue is extracted according to the preset time length through the preset sub-thread, and stored in the relational database, and a timestamp corresponding to the stored data is added.

6. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S4, after traversing and taking out the data in the relational database in chronological order, the data in the relational database is first converted into a two-dimensional data structure in bytes through JSON deserialization. A buffer processing area is defined for the different byte conditions of the converted data. In the buffer processing area, a preset a bytes is used as a byte array, and the converted data is continued to be traversed. The part exceeding a bytes or less than a bytes is put into the buffer processing area and spliced ​​to the front of the next data to form a byte array.

7. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 6, characterized in that: In step S5, the converted data is grouped in the form of index positioning, with 5 bytes as a byte group number, and each group of data is CRC-checked and verified according to 5 bytes.

8. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 7, characterized in that: In step S5, for the byte array that fails the check, the byte array is reassembled by sliding one byte to continue calculating the CRC checksum until the checksum passes.

9. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S6, when comparing the data types of adjacent byte arrays in chronological order, if they are both voltage or current, it is determined that data is lost, the byte array arranged at the rear is discarded, and the next byte array is added until the data types of adjacent byte arrays are different.

10. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S7, the compared byte array is parsed into plain text data, and the plain text data is placed in a two-dimensional array in the order of parsing, consistent with the form of the original data. When the plain text data in the two-dimensional array is complete original data, the plain text data is matched with the timestamp added in step S3 and stored in the database.

Images