Netty-based national standard intermodulation data analysis and storage method

Abstract

The invention discloses a netty-based national standard intermodulation data analysis and storage method, which comprises the following steps that: S1, a front-end front-end processor establishes communication with a server; S2, the server receives data and verifies the data; the method comprises the following steps: S1, transmitting data to a server side, S2, transmitting the data to the server side, S3, after the data is completely transmitted, verifying the data integrity by the server side, S4, after the verification of the data integrity is passed, reading a data stream in netty and converting the data stream into a byte [] array for data analysis, and S5, before the data is stored in a database, firstly verifying whether the data is duplicated data or not. According to the scheme provided by the invention, the development difficulty is reduced through the simplified API, and the high-performance network communication capability is provided. Memory management is optimized through the technologies of a memory pool, an object pool and the like, wide protocols are supported, custom expansion is allowed, stability and reliability are achieved at the same time, and benefits from active communities and detailed documents are achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of data analysis, and specifically is a method for analyzing and storing national standard intermodulation data based on Netty. Background Art

[0002] The introduction of netty as a background technology has brought a series of advantages and significantly improved the efficiency and experience of network programming. Netty's API design is simple and intuitive, which lowers the learning threshold, while providing powerful functional support, including built-in support for multiple codecs and protocols. Its high-performance design, such as memory pools and optimized thread models, ensures superior performance when handling high-concurrency connections. Netty also fixes known bugs in JDK N IO, provides a mature and stable framework, and thanks to active community support, can respond to bug fixes and function updates in a timely manner to meet the commercial application needs of different industries.

[0003] In the prior art, for example, the technical solution described in the patent with publication number CN114357209: a wizard-type national standard data modification operation method, computer equipment and storage medium, through which the wizard-type national standard data modification operation method is used to realize the visual operation of data processing, including the inductive processing of national standard data and military standard data, and the name of each processing item is displayed in the operation wizard interface, and the corresponding processing item is executed according to the operator's selection, by obtaining the class name of the command item in the configuration file, obtaining the command item through the class name, and then executing the command item to complete the execution of the processing item.

[0004] In the prior art, due to the complexity of Java NIO's API, multi-threaded management and platform dependency issues. Although Java NIO provides non-blocking I / O operations, its complicated class library and API increase the learning cost and require developers to master multiple core components. In addition, since NIO programming involves the Reactor mode, it is necessary to be familiar with Java multi-threaded programming, which increases the difficulty of programming. At the same time, some platform-related issues, such as epoll bugs, may lead to a huge waste of CPU resources and affect program performance. Summary of the invention

[0005] The purpose of the present invention is to provide a method for parsing and storing national standard intermodulation data based on Netty, so as to solve the problems of Java NIO in the prior art proposed in the background technology, that is, the Java NIO has complex APIs, requires multi-threaded management and platform dependence.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for parsing and storing national standard intermodulation data based on Netty, comprising the following steps:

[0008] Step S1, the front-end processor establishes communication with the server, and transmits a heartbeat to the server through the client. After the server receives the heartbeat, a connection is established;

[0009] Step S2: The server receives the data, verifies it, and returns the verification result to the client;

[0010] Step S3, when the data is completely transmitted, the server verifies the data integrity through CRC and a custom data decoder, and uses a custom decoder in netty to verify the integrity of the data;

[0011] Step S4, when the integrity of the data is verified, the data stream in Netty is converted into a byte[] array for data parsing, and the data in the array is converted and calculated in hexadecimal.

[0012] Step S5, before saving to the database, first verify whether the data is duplicate data; if correct, save it to the database, if it already exists, do not store it; after successful storage, reply to the front-end processor that the data has been received to avoid the front-end processor from sending data repeatedly.

[0013] According to the above technical solution, in step S2, the client also needs to determine whether the server has provided feedback, specifically:

[0014] If the client receives feedback from the server, it will proceed to the next data transmission process; if the client does not receive feedback, it will send a heartbeat again; if the number of heartbeats sent is greater than or equal to three, a heartbeat packet will be sent to test the server status; if the client fails to send data, the time point of the transmission failure should be recorded, and an attempt should be made to re-establish a new connection with the server at intervals of no less than 20s; when re-establishing a new connection, the data at the most recent time point should be transmitted first; and then the data after the failure time point will be re-transmitted.

[0015] According to the above technical solution, in step S3, verifying the integrity of the data includes the following steps:

[0016] Step S301, obtaining the readable length of the data packet;

[0017] Step S302, marking the header of the data packet to facilitate re-reading the data after all the data are read;

[0018] Step S303, all readable data are converted into a byte array, and the array pointer points to the last data;

[0019] Step S304, resetting the current pointer to the packet header to re-read the received data until all the data is received;

[0020] Step S305, determining whether the length of the data packet is correct with the data length provided in the data packet; if the data length is incorrect, continue to wait for data to be received until it is complete;

[0021] Step S306, determining whether the data packet header is the agreed byte; if it is not the agreed byte, disconnecting the current link.

[0022] According to the above technical solution, in step S4, the conversion and calculation of data includes heartbeat data and intermodulation data packets.

[0023] According to the above technical solution, the calculation of heartbeat data is specifically as follows:

[0024] Step S4011, when the current data is parsed as heartbeat data, the heartbeat status in the data is obtained to determine whether the device is online, and an assembling response is sent back to the front-end processor to indicate that the heartbeat data has been received;

[0025] Step S4012, after obtaining the device status, the device online status is sent through websocket instant communication to achieve real-time presentation of the device status.

[0026] According to the above technical solution, the calculation of the intermodulation data packet is specifically as follows:

[0027] Step S4021, when the data packet is of the transmission and feedback type or the data integrity type, the traffic survey data and image data therein are obtained, assembled into customized traffic data and then stored;

[0028] Step S4022, when the data packet is of the data integrity class, and the parsed data packet is the response data of the feedback information data packet and the retransmission data packet, it is assembled into a data type recognizable by the front-end processor and sent to the front-end processor;

[0029] Step S4023, when the data packet is response data, the server parses the data packet as a response instruction to determine the front-end processor status and current server information, and returns the information to the front-end processor;

[0030] In step S4024, the server can configure the modification instruction by itself and send it to the front-end processor. After the server modifies the instruction, the client receives the instruction and makes a modification action, and then transmits a reply to the server via a response data packet to indicate whether the modification is successful.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The solution in this invention reduces the development difficulty through simplified APIs and provides high-performance network communication capabilities, especially in high-concurrency scenarios. It optimizes memory management through technologies such as memory pools and object pools, supports a wide range of protocols and allows custom extensions, while having stability and reliability, thanks to its active community and detailed documentation. Netty also has strong security support and has been widely used and verified in multiple industries, ensuring its excellent performance and practicality as a network programming framework. Unified use of standardized decoding methods and data communication protocols. It can be used in different network environments, easy to use, and quick to get started. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the process of establishing a connection between the front-end processor and the server of the present invention;

[0034] Figure 2 The figure is a schematic diagram of the process of data storage in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Embodiment 1

[0037] like Figure 1 As shown, a method for parsing and storing national standard intermodulation data based on Netty includes the following steps:

[0038] Step S1, the front-end processor establishes communication with the server, and transmits a heartbeat to the server through the client. After the server receives the heartbeat, a connection is established;

[0039] Step S2: The server receives the data, verifies it, and returns the verification result to the client;

[0040] Step S3, such as Figure 2 As shown, after the data is completely transmitted, the server verifies the data integrity through CRC and a custom data decoder. A custom decoder is used in Netty to verify the integrity of the data.

[0041] Step S4, when the integrity of the data is verified, the data stream in Netty is converted into a byte[] array for data parsing, and the data in the array is converted and calculated in hexadecimal.

[0042] Step S5, before saving to the database, first verify whether the data is duplicate data; if correct, save it to the database, if it already exists, do not store it; after successful storage, reply to the front-end processor that the data has been received to avoid the front-end processor from sending data repeatedly.

[0043] The solution in this invention reduces the development difficulty through simplified APIs and provides high-performance network communication capabilities, especially in high-concurrency scenarios. It optimizes memory management through technologies such as memory pools and object pools, supports a wide range of protocols and allows custom extensions, while having stability and reliability, thanks to its active community and detailed documentation. Netty also has strong security support and has been widely used and verified in multiple industries, ensuring its excellent performance and practicality as a network programming framework. Unified use of standardized decoding methods and data communication protocols. It can be used in different network environments, easy to use, and quick to get started.

[0044] Embodiment 2

[0045] This embodiment is a further refinement of the first embodiment.

[0046] In step S2, Figure 1 As shown, the client also needs to determine whether the server has responded, specifically:

[0047] If the client receives feedback from the server, it will proceed to the next data transmission process; if the client does not receive feedback, it will send a heartbeat again; if the number of heartbeats sent is greater than or equal to three, a heartbeat packet will be sent to test the server status; if the client fails to send data, the time point of the transmission failure should be recorded, and an attempt should be made to re-establish a new connection with the server at intervals of no less than 20s; when re-establishing a new connection, the data at the most recent time point should be transmitted first; and then the data after the failure time point will be re-transmitted.

[0048] In step S3, verifying the integrity of the data includes the following steps:

[0049] Step S301, obtaining the readable length of the data packet;

[0050] Step S302, marking the header of the data packet to facilitate re-reading the data after all the data are read;

[0051] Step S303, all readable data are converted into a byte array, and the array pointer points to the last data;

[0052] Step S304, resetting the current pointer to the packet header to re-read the received data until all the data is received;

[0053] Step S305, determining whether the length of the data packet is correct with the data length provided in the data packet; if the data length is incorrect, continue to wait for data to be received until it is complete;

[0054] Step S306, determining whether the data packet header is the agreed byte; if it is not the agreed byte, disconnecting the current link.

[0055] In step S4, the conversion and calculation of data includes heartbeat data and intermodulation data packets.

[0056] The calculation of heartbeat data is as follows:

[0057] Step S4011, when the current data is parsed as heartbeat data, the heartbeat status in the data is obtained to determine whether the device is online, and an assembling response is sent back to the front-end processor to indicate that the heartbeat data has been received;

[0058] Step S4012, after obtaining the device status, the device online status is sent through websocket instant communication to achieve real-time presentation of the device status.

[0059] The calculation of the intermodulation data packet is as follows:

[0060] Step S4021, when the data packet is of the transmission and feedback type or the data integrity type, the traffic survey data and image data therein are obtained, assembled into customized traffic data and then stored;

[0061] Step S4022, when the data packet is of the data integrity class, and the parsed data packet is the response data of the feedback information data packet and the retransmission data packet, it is assembled into a data type recognizable by the front-end processor and sent to the front-end processor;

[0062] Step S4023, when the data packet is response data, the server parses the data packet as a response instruction to determine the front-end processor status and current server information, and returns the information to the front-end processor;

[0063] In step S4024, the server can configure the modification instruction by itself and send it to the front-end processor. After the server modifies the instruction, the client receives the instruction and makes a modification action, and then transmits a reply to the server via a response data packet to indicate whether the modification is successful.

[0064] Embodiment 3

[0065] This embodiment provides a specific implementation method, including the following steps:

[0066] Step 1: The front-end establishes communication with the server and transmits the heartbeat to the server through the client. The server establishes a connection after receiving the heartbeat.

[0067] Step 2: The server receives the data and verifies it, and returns the verification result to the client. If the client receives feedback from the server, it will proceed with the next data transmission process; if it does not receive feedback, it will send again; if the number of transmissions is greater than or equal to three times, it will send a heartbeat packet to test the server status; if the client fails to send data, it should record the time point of the transmission failure and try to re-establish a new connection with the server at an interval of not less than 20 seconds; when re-establishing a new connection, it should give priority to transmitting the data at the most recent time point; and then re-transmit the data after the failure time point.

[0068] Step 3: After the data is completely transmitted, the server verifies the data integrity by using CRC and a custom data decoder. A custom decoder is used in Netty to verify the data integrity.

[0069] 3.1, Get the readable length of the data packet.

[0070] 3.2, mark the header of the data packet to facilitate rereading the data after reading all the data.

[0071] 3.3, all readable data are converted into byte arrays, and the array pointer will point to the last data.

[0072] 3.4, reset the current pointer to the packet header to re-read the received data until all the data is received.

[0073] 3.5, determine whether the data packet length is correct with the data length provided in the data packet; if the data length is incorrect, continue to wait for the data to be received until it is complete.

[0074] 3.6, determine whether the data packet header is the agreed byte; if it is not the agreed byte, disconnect the current link.

[0075] 3.7, after receiving all the data packets, call the channel Read method in netty to parse the data packet.

[0076] Step 4: After the integrity of the data is verified, the data stream in Netty is converted into a byte[] array for data parsing. The data in the array is converted and calculated in hexadecimal.

[0077] 4.1, heartbeat data.

[0078] 4.1.1, when the current data is parsed as heartbeat data, obtain the heartbeat status in the data, determine whether the device is online, and assemble a response to reply that the front-end has received the heartbeat data.

[0079] 4.1.2, after obtaining the device status, send the device online status through websocket instant communication to achieve real-time presentation of the device status.

[0080] 4.2, intermodulation data packet.

[0081] 4.2.1. When the data packet is of "transmission and feedback type" or "data integrity type", obtain the traffic survey data and image data therein, assemble them into customized traffic data and then store them in the database.

[0082] 4.2.2, when the data packet is of "data integrity class" and the parsed data packet is "feedback information data packet", "retransmission data packet" and other response data, it is assembled into a data type recognizable by the front-end machine and sent to the front-end machine.

[0083] 4.2.3, when the data packet is "response data", the server parses the data packet as "response command" to determine the front-end machine status and current server information, and returns the information to the front-end machine.

[0084] 4.2.4, the server can configure the modification instructions by itself and send them to the front-end processor, such as "modify IP instruction", "client clock setting instruction", etc.; and after the server modifies the instruction, the client receives the instruction and makes the modification action, and then passes it to the server through the response data packet: a reply on whether the modification is successful.

[0085] Step 5: Before saving to the database, first verify whether the data is duplicate data. If it is correct, save it to the database. If it already exists, do not store it. After the storage is successful, reply to the front-end processor that the data has been received to avoid the front-end processor from sending data repeatedly.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0087] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for parsing and storing national standard intermodulation data based on Netty, characterized in that: The following steps are involved: Step S1, the front-end processor establishes communication with the server, and transmits a heartbeat to the server through the client. After the server receives the heartbeat, a connection is established; Step S2: The server receives the data, verifies it, and returns the verification result to the client; Step S3, when the data is completely transmitted, the server verifies the data integrity through CRC and a custom data decoder, and uses a custom decoder in netty to verify the integrity of the data; Step S4, when the integrity of the data is verified, the data stream in Netty is converted into a byte[] array for data parsing, and the data in the array is converted and calculated in hexadecimal. Step S5, before saving to the database, first verify whether the data is duplicate data; if correct, save it to the database; after successful storage, reply to the front-end front-end processor that the data has been received to avoid the front-end front-end processor from sending data repeatedly.

2. According to a method for parsing and storing national standard intermodulation data based on Netty in claim 1, it is characterized in that: In step S2, the client also needs to determine whether the server has responded, specifically: If the client receives feedback from the server, it will proceed to the next data transmission process; if the client does not receive feedback, it will send a heartbeat again; if the number of heartbeats sent is greater than or equal to three, it will send a heartbeat packet to test the server status; If the client fails to send data, it should record the time of transmission failure and try to re-establish a new connection with the server at an interval of not less than 20s. When re-establishing a new connection, it should give priority to transmitting data at the most recent time point, and then re-transmit the data after the failure time point.

3. According to a method for parsing and storing national standard intermodulation data based on Netty in claim 1, it is characterized in that: In step S3, verifying the integrity of the data includes the following steps: Step S301, obtaining the readable length of the data packet; Step S302, marking the header of the data packet to facilitate re-reading the data after all the data are read; Step S303, all readable data are converted into a byte array, and the array pointer points to the last data; Step S304, resetting the current pointer to the packet header to re-read the received data until all the data is received; Step S305, determining whether the length of the data packet is correct with the data length provided in the data packet; if the data length is incorrect, continue to wait for data to be received until it is complete; Step S306, determining whether the data packet header is the agreed byte; if it is not the agreed byte, disconnecting the current link.

4. According to a method for parsing and storing national standard intermodulation data based on Netty in claim 1, it is characterized in that: In step S4, the conversion and calculation of data includes heartbeat data and intermodulation data packets.

5. According to a method for parsing and storing national standard intermodulation data based on Netty according to claim 4, it is characterized in that: The calculation of heartbeat data is as follows: Step S4011, when the current data is parsed as heartbeat data, the heartbeat status in the data is obtained to determine whether the device is online, and an assembling response is sent back to the front-end processor to indicate that the heartbeat data has been received; Step S4012, after obtaining the device status, the device online status is sent through websocket instant communication to achieve real-time presentation of the device status.

6. According to a method for parsing and storing national standard intermodulation data based on Netty according to claim 5, it is characterized in that: The calculation of the intermodulation data packet is as follows: Step S4021, when the data packet is of the transmission and feedback type or the data integrity type, the traffic survey data and image data therein are obtained, assembled into customized traffic data and then stored; Step S4022, when the data packet is of the data integrity class, and the parsed data packet is the response data of the feedback information data packet and the retransmission data packet, it is assembled into a data type recognizable by the front-end processor and sent to the front-end processor; Step S4023, when the data packet is response data, the server parses the data packet as a response instruction to determine the front-end processor status and current server information, and returns the information to the front-end processor; In step S4024, the server can configure the modification instruction by itself and send it to the front-end processor. After the server modifies the instruction, the client receives the instruction and makes a modification action, and then transmits a reply to the server via a response data packet to indicate whether the modification is successful.

Images