A processing method for coping with a large amount of data gushing out in milliseconds through a serial port

Abstract

The present invention belongs to the field of power systems and provides a processing method for coping with a large amount of millisecond-level data surging out of a serial port; the method includes the following steps: S1, determining the specifications of the serial port, the serial port cable and the serial port cable chip; S2, the serial port receives the original data and stores it in the buffer area, and the stack area obtains data from the buffer area in real time; S3, regularly extracting data for a preset duration from the stack area and storing it in a relational database, adding a timestamp; S4, taking out the data in chronological order, setting up a buffer processing area, and performing byte array splicing; S5, performing a CRC checksum; S6, comparing the data types of adjacent byte arrays; S7, obtaining the byte arrays after comparison in step S6, parsing them into plaintext data, and storing them in the database. The present invention can ensure the integrity of the data and improve the data processing efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of power systems, and particularly relates to a processing method for coping with a large amount of millisecond-level data surging out of a serial port. Background Art

[0002] In a power system, an RS485 network is used to implement communication between electric energy meters and a meter reading concentrator. The meter reading concentrator collects data of multiple electric energy meters through the RS485 network and uploads it to the server of the power management department, realizing remote automatic meter reading, improving the efficiency and accuracy of meter reading, and reducing labor costs. In the field of industrial data communication, the serial port of the RS485 network and related devices follow standard specifications, and the baud rate supported during its normal operation is set to 115200. However, with the continuous increase in the monitoring data traffic during the system operation, especially when the amount of data to be received within 1 second reaches 25.6 KB, the transmission capacity of the preset baud rate can no longer fully meet the actual data transmission requirements. This mismatch causes the serial port to become blocked, and the data transmission efficiency of the serial port immediately shows a rapid decline. This inevitably leads to a large number of byte losses during the communication process, which in turn seriously affects the accuracy, integrity of the entire data communication, as well as the stability and reliability of the system. It may cause subsequent data processing, analysis, and related control operations relying on this data communication link to be unable to be carried out normally and effectively, and may even trigger a series of chain problems such as partial function failure or serious performance degradation of the entire monitoring and control system. Moreover, since individual threads are involved in database persistence operations, which may have latency problems, caches should be used to solve the latency problems at places where latency may occur, and the execution between threads does not affect each other and is not blocked, thereby improving the overall efficiency of software processing.

[0003] Patent document CN201810286486.8 discloses a reliable power line carrier polling communication method, which uses protocol frames as carriers and includes the following key steps: designing the protocol format of the network layer (composed of multiple parts such as host address, slave address, etc.), first verifying the data, then sequentially judging the frame sequence number and the protocol frame status bit, and according to the status bit situation (when it is 0), letting the slave judge whether there is data in the protocol frame sent to the host last time, and timeout processing steps. This method can reduce the packet loss rate of the communication system, achieve multi-point high-speed reliable polling communication on the power line in an industrial harsh channel, have good QoS guarantee, meet the high reliability requirements of information collection, and reduce the network failure rate.

[0004] However, the existing technologies have at least the following technical problems: The existing technologies do not conduct research targeting RS485 networks as the application object, so there are differences in application implementation. The existing technologies only solve the problem of byte loss from the aspects of protocol and data frame verification, and do not systematically optimize the communication failures caused by the rapid transmission of large-capacity data from the aspects of transmission medium, data receiving method, and data verification method, resulting in the inability to complete accurate transmission when the communication traffic suddenly increases. Summary of the Invention

[0005] The present invention aims to address the technical problems existing in the prior art and provides a processing method for coping with a large amount of data gushing out of a serial port in milliseconds. This method can ensure the integrity of the data received by the serial port, build an efficient data caching mechanism through caching technology. At the moment when the serial port receives data, the data can be quickly placed in the designed buffer area, avoiding the stagnation time caused by waiting for the serial port to transmit each byte one by one, and improving the system data processing efficiency.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] A processing method for coping with a large amount of data gushing out of a serial port in milliseconds, comprising the following steps:

[0008] S1. Debug the serial port, switch the baud rate of the serial port required for data transmission in milliseconds, and then determine the serial port specification. Determine the serial port line and the serial port line chip according to the serial port specification;

[0009] S2. The serial port receives the original data and stores it in the buffer area. The stack area obtains data from the buffer area in real time and deletes the obtained data at the same time;

[0010] S3. Start the cyclic extraction method, regularly extract data for a preset duration from the stack area, serialize it and store it in the queue. At the same time, extract and store the data in the queue into a relational database and add a timestamp;

[0011] S4. Traverse and retrieve the data in the relational database in chronological order, convert the data into byte data, and set a buffer processing area. Concatenate the byte data in the buffer processing area;

[0012] S5. Verify the byte data in the byte array through CRC checksum, and at the same time judge whether the data type of the current byte array is voltage or current;

[0013] S6. Obtain the byte array and data type that pass the CRC checksum, compare the data types of adjacent byte arrays in chronological order. If they are both voltage or current, discard the byte array arranged behind until the data types of the byte arrays are different;

[0014] S7. Obtain the byte array after comparison in step S6, parse it into plaintext data, and store it in the database.

[0015] Optionally, in step S1, use the baud rate configuration derivation method to calculate the inductive reactance of the serial communication line, and then determine the serial port specification. The calculation process of the inductive reactance of the serial communication line is as follows:

[0016] Calculate the inductance of the round-section straight wire of the serial communication line. The calculation formula is as follows:

[0017] ,

[0018] Where, is the inductance of the round-section straight wire of the serial communication line, is the permeability of free space, is the wire length, is the wire radius;

[0019] Calculate the inductive reactance of the serial communication line. The formula is as follows:

[0020] ,

[0021] Where, is the inductive reactance of the serial communication line, is the frequency.

[0022] Optionally, in step S2, send the original data to the selected serial port. The original data is in the form of a two-dimensional array. The serial port includes a buffer area and a stack area. The serial port receives the data and stores the received data in the buffer area. The stack area retrieves the data from the buffer area in real time and deletes the retrieved data at the same time.

[0023] Optionally, in step S3, serialize the data extracted for a preset duration through the JSON serialization method, convert the data for the preset duration into a string, and store it in the queue. At the same time, delete the data for the preset duration that has been extracted from the stack area.

[0024] Optionally, in step S3, when the loop extraction method is enabled, extract the data in the queue according to the preset duration through a preset sub-thread, and store it in the relational database. At the same time, add a timestamp corresponding to the stored data.

[0025] Optionally, in step S4, after retrieving the data in the relational database in chronological order, first convert the data in the relational database into a two-dimensional data structure in bytes through JSON deserialization. For the converted data with different byte situations, define a buffer processing area. In the buffer processing area, use a preset a bytes as a byte array, and continue to traverse the converted data. For the part that exceeds a bytes or is less than a bytes, put it into the buffer processing area and splice it to the front of the next piece of data to form a byte array.

[0026] Optionally, in step S5, take 5 bytes as a byte group, and group the converted data in the form of index positioning. Each group of data is subjected to CRC checksum verification according to 5 bytes.

[0027] Optionally, in step S5, for the byte array that fails the verification, slide one byte and splice the byte array again to continue calculating the CRC checksum until the checksum passes.

[0028] Optionally, in step S6, when comparing the data types of adjacent byte arrays in chronological order, if they are both voltage or current, it is determined that data is lost, discard the byte array arranged later, and supplement the next byte array until the data types of adjacent byte arrays are different.

[0029] Optionally, in step S7, parse the compared byte array into plaintext data, put the plaintext data into a two-dimensional array in the parsed order, and keep it in the same form as the original data. When the plaintext data in the two-dimensional array is the complete original data, correspond the plaintext data with the timestamp added in step S3 and store it in the database.

[0030] Compared with the prior art, the beneficial effects produced by the present invention are:

[0031] (1) By presetting a child thread, the present invention realizes the combined use of multi-threading, caching technology, and stack buffering technology, ensuring the integrity of the serial port received data, avoiding serial port blockage, and thus shortening the time for subsequent data parsing and storage;

[0032] (2) During the dynamic process of the serial port of the present invention continuously receiving a large amount of data, the preset child thread can be dedicated to closely interacting with the port, accurately and real-time capturing the continuously incoming data stream, making it not affected by the operation delay of other software levels, minimizing the risk of data omission, and firmly ensuring the integrity of the data receiving link;

[0033] (3) The present invention constructs an efficient data caching mechanism through the caching technologies of the serial port buffer and the stack area. At the moment when the serial port receives data, the data can be quickly placed in the designed stack area. When the subsequent data parsing stage is entered, the multi-threaded architecture allows independent parsing threads to directly and quickly retrieve data from the stack area, avoiding the stagnation time caused by waiting for the serial port to transmit data byte by byte, greatly accelerating the parsing process, and comprehensively improving the data processing efficiency and reliability of the system. Description of the Drawings

[0034] Figure 1 It is a flowchart of a processing method for coping with a large amount of data gushing out at the millisecond level of the serial port in an embodiment of the present invention;

[0035] Figure 2 It is a technical roadmap of a processing method for coping with a large amount of data gushing out at the millisecond level of the serial port in an embodiment of the present invention. Detailed Embodiments

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0037] Embodiment 1

[0038] Combined with Figure 1 - Figure 2 As shown, the present invention provides a processing method for coping with a large amount of data gushing out at the millisecond level of the serial port, including the following steps:

[0039] S1. Debug the serial port, switch the serial port baud rate required under millisecond-level data transmission, and then determine the serial port specifications. According to the serial port specifications, determine the serial port cable and the serial port cable chip;

[0040] S2. The serial port receives the original data and stores it in the buffer area. The stack area continuously obtains data from the buffer area and deletes the obtained data at the same time;

[0041] S3. Start the cyclic extraction method, regularly extract data for a preset duration from the stack area, serialize it and store it in the queue. At the same time, extract and store the data in the queue into the relational database and add a timestamp;

[0042] S4. Traverse and retrieve the data in the relational database in chronological order, convert the data into byte data, and set a buffer processing area. In the buffer processing area, splice the byte data into a byte array;

[0043] S5. Verify the byte data in the byte array through CRC checksum, and at the same time determine whether the data type of the current byte array is voltage or current;

[0044] S6. Obtain the byte array and data type that pass the CRC checksum, compare the data types of adjacent byte arrays in chronological order. If they are both voltage or current, discard the byte array arranged later until the data types of the byte arrays are different;

[0045] S7. Obtain the byte array after comparison in step S6, parse it into plaintext data, and store it in the database.

[0046] Embodiment 2

[0047] Compared with Embodiment 1, in this embodiment, in step S1, since the inductive reactance of the serial communication line will affect the signal transmission ability, signal consumption and transmission distance, it is necessary to evaluate and debug the serial port according to the inductive reactance of the serial communication line to determine the serial port specifications.

[0048] As Figure 2 shown in the technical roadmap of, determining the serial port specifications, debugging the serial port, and switching the baud rate are specifically as follows: Assume that the initial specifications (initial baud rate) of the serial port are 19,200 bps. When the serial port specifications do not meet the requirement of a large amount of data gushing out in milliseconds, debug the serial port. At this time, the serial port specifications can be switched to a serial port with a baud rate of 1.5 Mbps to meet the communication requirements of a large amount of data gushing out in milliseconds.

[0049] Regarding the determination of the serial port baud rate, specifically, the inductive reactance of the serial communication line can be calculated by using the baud rate configuration derivation method, and then the serial port baud rate can be determined, and further the serial port specifications can be determined. The calculation process of the inductive reactance of the serial communication line is as follows:

[0050] Calculate the inductance of the round-section straight wire of the serial communication line. The calculation formula is as follows:

[0051] ,

[0052] Among them, is the inductance of the round-section straight wire of the serial communication line, is the magnetic permeability of vacuum, is the wire length, is the wire radius;

[0053] Calculate the inductive reactance of the serial communication line. The formula is as follows:

[0054] ,

[0055] Among them, is the inductive reactance of the serial communication line, is the frequency;

[0056] That is, the frequency The larger it is, the larger the inductive reactance is, and the stronger the ability to block signals.

[0057] For the RS485 communication used by the chip to transmit 26.4KB of data per second, after multiple serial port evaluation and debugging, it is finally determined to use a baud rate of 1.5Mbps for communication. The BL1590 high-speed chip is selected for the chip, and the serial port line of the relatively stable BL1590 chip is used as the communication line.

[0058] Further, in step S2, the original data is sent to the selected serial port. The original data is in the form of a two-dimensional array. The serial port includes a buffer area and a stack area. The serial port receives the data and stores the received data in the buffer area. The stack area obtains the data from the buffer area in real time and deletes the obtained data at the same time.

[0059] Further, in step S3, the data extracted for a preset duration is serialized by the JSON serialization method, the data for the preset duration is converted into a string, and stored in the queue. At the same time, the data for the preset duration that has been extracted from the stack area is deleted;

[0060] Further, when the loop extraction method is enabled, the data in the queue is extracted according to the preset duration by a preset sub-thread, and stored in the relational database. At the same time, a timestamp corresponding to the stored data is added.

[0061] Embodiment 3

[0062] Compared with Embodiment 1, in this embodiment, in step S4, after traversing and taking out the data in the relational database in chronological order, first convert the data in the relational database into a two-dimensional data structure in bytes through JSON deserialization. For the converted data with different bytes, a buffer processing area is defined. In the buffer processing area, a preset a bytes are used as a byte array, and the converted data is traversed continuously. For the part that exceeds a bytes or is less than a bytes, it is put into the buffer processing area and spliced to the front of the next piece of data to form a byte array.

[0063] As a preferred method, in step S5, 5 bytes are used as a byte group number, and the converted data is grouped in the form of index positioning. Each group of data is subjected to CRC checksum check according to 5 bytes. Among them, the CRC checksum is Cyclic Redundancy Check.

[0064] Further, in step S5, for the byte array that fails the check, the byte array is re-spliced by slipping one byte and the CRC checksum is calculated continuously until the checksum passes.

[0065] Embodiment 4

[0066] Compared with Embodiment 1, in this embodiment, after the serial port receives data, the data is saved in the serial port buffer area, and the stack area retrieves data from the buffer area in real time, without blocking the serial port from receiving data, so that the data received by the serial port can be saved to the stack area; after the data is retrieved, the data retrieved from the buffer area is deleted to ensure that the data in the buffer area will not overflow and ensure the continuous reception of the serial port;

[0067] Step S3 is specifically as follows: Enable the cyclic extraction method, regularly extract data for a preset duration from the stack area, serialize the data extracted for the preset duration through the JSON serialization method, convert the data for the preset duration into a string, and store it in the queue. At the same time, delete the data for the preset duration that has been extracted from the stack area. At the same time, extract the data in the queue at the preset duration through a preset sub-thread, store it in the relational database, and add a timestamp;

[0068] Among them, when enabling the cyclic extraction method, 3 seconds of data is cyclically retrieved from the stack area, the retrieved 3 seconds of data is JSON serialized into the queue, and then the data within 3 seconds retrieved from the stack area is deleted to ensure that the data in the stack area will not overflow. This method can ensure that the data saving speed is greater than 3 seconds, that is, greater than the serial port data reception speed;

[0069] The stack area solves the problem that data will be lost when the data volume is large and the data storage takes a long time, ensuring the integrity of data storage.

[0070] Step S4 is specifically as follows: Traverse and retrieve the data in the relational database in chronological order, and convert the data in the relational database into a two-dimensional data structure in bytes through JSON deserialization. Since the extracted original data is currently in the form of a byte two-dimensional array, it needs to be converted into a string through JSON deserialization;

[0071] Among them, the original data is stored in the relational database in the form of a byte two-dimensional array (note: the original two-dimensional array in C# is byte[][]. In the actual code, a more efficient linear list provided by.net will be used for storage);

[0072] For the converted data with different byte situations, define a buffer processing area. In the buffer processing area, a preset number of a bytes are used as a byte array, and continue to traverse the converted data. For the part that exceeds a bytes or is less than a bytes, put it into the buffer processing area and splice it to the front of the next piece of data to form a byte array;

[0073] Step S5 specifically is as follows: Taking 5 bytes as a group of byte numbers, group the converted data in the form of index positioning. Each group of data is subjected to CRC checksum verification according to 5 bytes. For the byte array that fails the verification, slide one byte and re - splice the byte array to continue calculating the CRC checksum until the checksum passes;

[0074] Meanwhile, according to the difference in the form of CRC checksum between voltage data and current data, determine whether the data type of the current byte array is voltage or current;

[0075] Step S6 specifically is as follows: Obtain the byte array and data type that pass the CRC checksum. When comparing the types of adjacent byte arrays in chronological order, if they are both voltage or current, it is determined that data is lost. Discard the byte array arranged later and supplement the next byte array until the types of adjacent byte data are different;

[0076] Among them, since current and voltage appear in pairs, after each calculation of data, it is necessary to compare with the previous piece of data. If the two pieces of data are both voltage or current, it means that there is data loss in the middle. At this time, this piece of data should be discarded and the calculation of the next piece of data should be continued until it is different from the last piece of data in terms of data type;

[0077] Step S7 specifically is as follows: Obtain the byte array after comparison in Step S6, parse the byte array after comparison into plain - text data, and put the plain - text data into a two - dimensional array in the order of parsing, keeping it consistent with the form of the original data. When the plain - text data in the two - dimensional array is the complete original data, correspond the plain - text data with the timestamp added in Step S3 and store it in the database.

[0078] The above - mentioned are only embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the scope of the application of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for processing a large amount of data flowing out of a serial port at the millisecond level, characterized in that: The following steps are involved: S1. Debug the serial port, switch the serial port baud rate required for millisecond-level data transmission, and then determine the serial port specifications, and determine the serial port line and serial port line chip according to the serial port specifications; S2, the serial port receives the original data and stores it in the buffer area, the stack area obtains data from the buffer area in real time, and deletes the obtained data at the same time; S3. Start the cyclic extraction mode, regularly extract the data of the preset time length from the stack area, serialize and store it in the queue, and extract and store the data in the queue into the relational database, and add a timestamp; S4, traversing and taking out the data in the relational database in time order, converting the data into byte data, and setting a buffer processing area, and performing byte array splicing on the byte data in the buffer processing area; S5, verifying the byte data in the byte array through CRC checksum, and determining whether the data type of the current byte array is voltage or current; S6, obtaining the byte array and data type that pass the CRC checksum, comparing the data types of adjacent byte arrays in chronological order, and if they are both voltage or current, discarding the byte array arranged at the back, until the data types of the byte arrays are different; S7, obtaining the byte array after comparison in step S6, parsing it into plain text data, and storing it in a database; In step S1, the inductive reactance of the serial communication line is calculated using the baud rate configuration derivation method, and then the serial port specifications are determined. The calculation process of the inductive reactance of the serial communication line is as follows: Calculate the inductance of the circular cross-section straight conductor of the serial communication line. The calculation formula is as follows: ， in, is the inductance of the circular cross-section straight conductor of the serial communication line, is the vacuum permeability, is the wire length, is the wire radius; Calculate the inductive reactance of the serial communication line, the formula is as follows: ， in, is the inductive reactance of the serial communication line, is the frequency.

2. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S2, the original data is sent to the selected serial port. The original data is in the form of a two-dimensional array. The serial port includes a buffer area and a stack area. The serial port receives data and stores the received data in the buffer area. The stack area obtains data from the buffer area in real time and deletes the obtained data at the same time.

3. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S3, the data of the preset duration is serialized by the JSON serialization method, the data of the preset duration is converted into a string, and stored in the queue, and the data of the preset duration that has been extracted from the stack area is deleted.

4. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 3, characterized in that: In step S3, when the cyclic extraction mode is started, the data in the queue is extracted according to the preset time length through the preset sub-thread, and stored in the relational database, and a timestamp corresponding to the stored data is added.

5. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S4, after traversing and taking out the data in the relational database in chronological order, the data in the relational database is first converted into a two-dimensional data structure in bytes through JSON deserialization. A buffer processing area is defined for the different byte conditions of the converted data. In the buffer processing area, a preset a bytes is used as a byte array, and the converted data is continued to be traversed. The part exceeding a bytes or less than a bytes is put into the buffer processing area and spliced ​​to the front of the next data to form a byte array.

6. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 5, characterized in that: In step S5, the converted data is grouped in the form of index positioning, with 5 bytes as a byte group number, and each group of data is CRC-checked and verified according to 5 bytes.

7. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 6, characterized in that: In step S5, for the byte array that fails the check, the byte array is reassembled by sliding one byte to continue calculating the CRC checksum until the checksum passes.

8. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S6, when comparing the data types of adjacent byte arrays in chronological order, if they are both voltage or current, it is determined that data is lost, the byte array arranged at the rear is discarded, and the next byte array is added until the data types of adjacent byte arrays are different.

9. The method for processing a large amount of data flowing out of a serial port at the millisecond level according to claim 1, characterized in that: In step S7, the compared byte array is parsed into plain text data, and the plain text data is placed in a two-dimensional array in the order of parsing, consistent with the form of the original data. When the plain text data in the two-dimensional array is complete original data, the plain text data is matched with the timestamp added in step S3 and stored in the database.

Images