Method and system for processing and receiving actually sent data

Abstract

The invention provides an actually sent data processing and receiving method and system, and the method comprises the steps: monitoring the TCP connection transmitted by a slave station in real time based on a master station, receiving a data frame transmitted by the slave station when the connection is monitored, and carrying out the analysis and verification of the data frame based on a preset protocol; after the verification is passed, pushing the target data obtained by analysis to a corresponding agent node in a Kafka message queue in real time, and sequentially storing the target data in each agent node to a relational database based on a pushing result; constructing a distributed search engine, and associating the distributed search engine with the relational database; and monitoring a data change event in the relational database in real time based on the association result, and synchronously indexing and analyzing the hot data corresponding to the data change event according to the distributed search engine based on the monitoring result. The real-time processing capability of the data acquisition and processing service is improved, the increase of the data volume and the increase of the page view can be easily dealt with, and the reliability and the consistency of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for processing and receiving real-time data. Background Art

[0002] At present, with the popularization of Internet of Things, big data and cloud computing technologies, the amount of data is growing explosively, which puts higher requirements on real-time processing and receiving capabilities;

[0003] Although real-time data processing and reception technology plays an important role in promoting digital transformation, it still faces many challenges;

[0004] First, with the surge in data volume and the increasing real-time requirements, ensuring the timeliness of data processing has become a major problem. The system needs to respond quickly and process massive amounts of data to support instant decision-making and feedback. However, current technologies may encounter performance bottlenecks in actual applications, affecting the ability to process data in real time.

[0005] Secondly, system scalability and stability are another major challenge. As the business scale grows, the data processing system must be able to flexibly expand to meet the changing needs. However, when facing high concurrency and large-scale data, existing technologies may encounter expansion difficulties, uneven resource allocation, or system crashes, which will affect business continuity and stability.

[0006] In addition, data consistency and accuracy are also issues that cannot be ignored. In a distributed system, data consistency and accuracy may be affected by various factors (such as network delays and node failures), which will not only reduce data quality but also may lead to wrong decisions and actions.

[0007] Therefore, in order to overcome the above-mentioned defects, the present invention provides a method and system for processing and receiving real-time data. Summary of the invention

[0008] The present invention provides a method and system for processing and receiving real-time data, which is used to monitor the TCP connection sent by the slave station through the master station, and when the connection exists, the data frame sent by the slave station is received, parsed and verified, thereby ensuring the accuracy and reliability of the data received by the master station. Secondly, after the verification is passed, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and then stored in the relational database, thereby ensuring the effective reception and storage of the data sent by the slave station, and improving the reliability of real-time data processing. Finally, a distributed search engine is constructed and associated with the relational database, so that when there is a data change event, the hot data corresponding to the data change event is indexed and analyzed in time, thereby improving the real-time processing capability of the data acquisition and processing service. At the same time, it can easily cope with the growth of data volume and the increase of access volume, ensuring the reliability and consistency of the data, and reducing the risk of data loss and error.

[0009] The present invention provides a method for processing and receiving real-time data, comprising:

[0010] Step 1: Based on the master station, the TCP connection sent by the slave station is monitored in real time, and when the connection is detected, the data frame sent by the slave station is received, and the data frame is parsed and verified based on the preset protocol;

[0011] Step 2: After verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is stored in the relational database in turn based on the push results;

[0012] Step 3: Build a distributed search engine and associate the distributed search engine with a relational database;

[0013] Step 4: Based on the correlation results, the data change events in the relational database are monitored in real time, and based on the monitoring results, the hot data corresponding to the data change events are synchronously indexed and analyzed using a distributed search engine.

[0014] Preferably, a method for processing and receiving real-time data, in step 1, based on the master station real-time monitoring of the TCP connection sent by the slave station, and when the connection is detected, receiving the data frame sent by the slave station, including:

[0015] Get the available communication port of the master station and create a TCP listening socket on the available communication port;

[0016] Based on the creation result, the TCP connection sent by the slave is monitored, and when the TCP connection is detected, a connection socket between the master and the slave is established;

[0017] Based on the construction result, the slave station is controlled to perform protocol encapsulation on the data frame based on the preset protocol, and the protocol encapsulated data frame is sent to the master station based on the connection socket.

[0018] Preferably, a method for processing and receiving real data, in step 1, parsing and verifying the data frame based on a preset protocol, includes:

[0019] Obtain a preset protocol, extract a format frame in the preset protocol, and parse the data frame based on the format frame to obtain multi-dimensional basic information contained in the data frame;

[0020] Extracting the start character of the identification frame and the standard length of the data frame from the multi-dimensional basic information, and performing length detection on the data frame based on the start character of the identification frame to obtain the actual received length;

[0021] Performing a first check on the data frame based on the standard length and the actual received length of the data frame, and locating the control field of the data frame from the multi-dimensional basic information based on the parsing result of the data frame by the format frame after the first check passes;

[0022] Perform a global traversal of the fields in the control domain, and obtain the target value corresponding to each field based on the global traversal results;

[0023] Determine the function indication of each field according to the field and the target value based on the preset protocol, extract multi-dimensional feature parameters of the data frame based on the function indication, and perform a second check on the extracted multi-dimensional feature parameters based on the function indication;

[0024] After the second check is passed, the information object address of the data frame is extracted from the multi-dimensional basic information based on the parsing result of the format frame on the data frame, and the predefined data model is matched from the preset model library based on the information object address;

[0025] Parse the data frame based on the predefined data model, determine the semantic features and business features corresponding to the data frame, and extract the data value of the data frame;

[0026] Segment the data frame based on a natural language system to obtain multiple target words, and determine the part of speech of each target word based on the natural language system;

[0027] Perform content verification on the data frame based on semantic features, business features and the part of speech of each target time;

[0028] After the content verification is passed, the target data content in the data frame is format converted, and the parsing and verification results of the data frame are recorded based on the format conversion result to obtain a log file.

[0029] Preferably, a method for processing and receiving real data, in step 1, parsing and verifying the data frame based on a preset protocol, includes:

[0030] Obtain the parsing and verification links of the data frame, and determine the corresponding processing plan when each parsing and verification link fails based on management requirements;

[0031] A multi-touch jump mechanism is constructed based on the analysis and verification links, and the corresponding processing scheme when each analysis and verification link fails is associated and bound with the multi-touch jump mechanism;

[0032] Based on the association binding result, the result status of each analysis and verification link is received in real time, and when the real-time received result is judged as not passed, a burst response is made to the current analysis and verification link based on the multi-touch jump mechanism;

[0033] Based on the emergency response results, emergency processing is performed on the current analysis and verification links according to the processing plan.

[0034] Preferably, a method for processing and receiving real-time data, in step 2, after verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is sequentially stored in the relational database based on the push result, including:

[0035] Obtain the structural distribution characteristics of each proxy node in the Kafka message queue, and determine the storage path between each proxy node in the Kafka message queue for the data to be stored based on the structural distribution characteristics;

[0036] At the same time, extracting basic attributes of each proxy node, and determining a rated operating state of each proxy node based on the basic attributes, and determining a resource allocation amount for each proxy node based on the rated operating state;

[0037] Configure the Kafka message queue based on the storage path and resource allocation, and extract the data topic of each proxy node in the Kafka message queue based on the configuration result. At the same time, convert the parsed target data into key-value pairs based on the operation requirements of the Kafka message queue;

[0038] Match the key in the key-value pair with the data subject of each proxy node, and determine the proxy node corresponding to different target data based on the matching result;

[0039] Based on the storage path, the values ​​in the key-value pair are pushed to the corresponding proxy nodes in the Kafka message queue in real time, and the data sequence of each proxy node is obtained;

[0040] Obtain the API interface of the relational database, and store the data sequence of each proxy node in the Kafka message queue to the relational database based on the API interface.

[0041] Preferably, a method for processing and receiving real-time data pushes the values ​​in the key-value pairs to the corresponding proxy nodes in the Kafka message queue in real time based on the storage path, including:

[0042] Obtain target push content based on the value in the key-value pair, and split the target push content into N content blocks;

[0043] Traverse the characters in each content block in turn, obtain redundant information in each content block, and remove the redundant information;

[0044] Compress the content blocks after redundant information is removed based on a preset compression method, and push the compressed content blocks to the corresponding proxy nodes in the Kafka message queue in real time;

[0045] The storage space status of each proxy node is periodically monitored based on the push results, and when the storage space status of the proxy node meets the preset cleaning conditions, the deletion data of the stored data in each proxy node is determined based on the time log, and the deleted data is eliminated.

[0046] Preferably, a method for processing and receiving real-time data, in step 3, constructing a distributed search engine and associating the distributed search engine with a relational database, includes:

[0047] Obtaining a framework structure of a relational database, and determining a node layout when searching for data in the relational database based on the framework result, and determining a communication address of each node in the relational database based on the node layout;

[0048] Obtain the index source of each node based on the communication address, and build the local index of each node based on the index source;

[0049] Adding the local index to the script file of the target search engine for the first time, and obtaining a distributed search engine based on the result of the first addition;

[0050] The distributed search engine is interfaced with the relational database, and an identifier of the relational database is obtained based on the interface connection result;

[0051] The identifier is added to the script file of the target search engine for a second time, and the association between the distributed search engine and the relational database is completed based on the second addition.

[0052] Preferably, a method for processing and receiving real-time data, in step 4, real-time monitoring of data change events in a relational database based on the correlation results, and synchronous indexing and analysis of hot data corresponding to the data change events according to a distributed search engine based on the monitoring results, including:

[0053] Monitor the status of stored data in the relational database in real time, and start the distributed search engine synchronously when data change events are detected;

[0054] Based on the synchronization start result, the distributed search engine is controlled to search for the data change node, and the hot data corresponding to the data change event is determined based on the search result;

[0055] Hot data is indexed based on a distributed search engine, and the indexed hot data is analyzed and processed.

[0056] The present invention provides a system for processing and receiving real data, comprising:

[0057] The data receiving and parsing module is used to monitor the TCP connection sent by the slave station in real time based on the master station, and when the connection is detected, receive the data frame sent by the slave station, and parse and verify the data frame based on the preset protocol;

[0058] The data storage module is used to push the parsed target data to the corresponding proxy node in the Kafka message queue in real time after verification, and store the target data in each proxy node in the relational database in turn based on the push results;

[0059] The association module is used to build a distributed search engine and associate the distributed search engine with a relational database;

[0060] The monitoring management module is used to monitor data change events in the relational database in real time based on the correlation results, and synchronously index and analyze the hot data corresponding to the data change events according to the distributed search engine based on the monitoring results.

[0061] Preferably, a system for processing and receiving real data is characterized in that the data receiving and parsing module includes:

[0062] A monitoring mechanism creation unit is used to obtain an available communication port of the master station and create a TCP monitoring socket on the available communication port;

[0063] A connection unit, for monitoring the TCP connection sent by the slave station based on the creation result, and building a connection socket between the master station and the slave station when the TCP connection is detected;

[0064] The data receiving unit is used to control the slave station to perform protocol encapsulation on the data frame based on the preset protocol based on the construction result, and send the protocol encapsulated data frame to the master station based on the connection socket.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. The master station monitors the TCP connection sent by the slave station, and when a connection exists, receives, parses and verifies the data frames sent by the slave station, ensuring the accuracy and reliability of the data received by the master station. Secondly, after the verification is passed, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and then stored in the relational database, ensuring the effective reception and storage of the data sent by the slave station, and improving the reliability of real-time data processing. Finally, a distributed search engine is built and associated with the relational database, so that when there is a data change event, the hot data corresponding to the data change event can be indexed and analyzed in time, improving the real-time processing capability of the data acquisition and processing service. At the same time, it can easily cope with the growth of data volume and the increase in access volume, ensuring the reliability and consistency of the data, and reducing the risk of data loss and errors.

[0067] 2. By creating a TCP listening socket on the available communication port of the master station, it is convenient to monitor the TCP connection sent by the slave station in real time. It is also convenient to accurately and reliably build a connection socket between the master station and the slave station when a TCP connection exists, providing a channel and convenience for data transmission. Secondly, after building the connection socket, the data frame of the slave station is encapsulated according to the preset protocol, and the encapsulated data frame is sent to the master station through the connection socket, ensuring the reliability of the master station's data reception.

[0068] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0069] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0071] Figure 1 This is a flow chart of a method for processing and receiving real data in an embodiment of the present invention;

[0072] Figure 2This is a flowchart of step 1 in a method for processing and receiving real data in an embodiment of the present invention;

[0073] Figure 3 The present invention is a structural diagram of a system for processing and receiving real data in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0075] Embodiment 1:

[0076] This embodiment provides a method for processing and receiving real data. Figure 1 As shown, including:

[0077] Step 1: Based on the master station, the TCP connection sent by the slave station is monitored in real time, and when the connection is detected, the data frame sent by the slave station is received, and the data frame is parsed and verified based on the preset protocol;

[0078] Step 2: After verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is stored in the relational database in turn based on the push results;

[0079] Step 3: Build a distributed search engine and associate the distributed search engine with a relational database;

[0080] Step 4: Based on the correlation results, the data change events in the relational database are monitored in real time, and based on the monitoring results, the hot data corresponding to the data change events are synchronously indexed and analyzed using a distributed search engine.

[0081] In this embodiment, the master station and the slave station are respectively a data receiving end and a data sending end, and are terminals for performing data reception and transmission.

[0082] In this embodiment, the TCP connection is a connection-oriented, reliable network communication connection. In the TCP connection, the communicating parties establish a connection through a three-way handshake before transmitting data to ensure that both parties are ready for communication. After the connection is established, data can be transmitted sequentially and reliably, and it has mechanisms such as flow control and error control to ensure the quality of data transmission. When the data transmission is completed, the connection is closed through specific steps.

[0083] In this embodiment, the data frame refers to data sent from the slave station to the master station, including data and necessary control information, and is used for data transmission and communication on the link layer.

[0084] In this embodiment, the preset protocol is known in advance, and may be, for example, IEC 60870-5-104 (abbreviated as 104 protocol).

[0085] In this embodiment, parsing and verifying the data frame based on the preset protocol may be parsing and verifying the start character, length, control field, address field, information object address (IOA) of the identification frame and the actual data content, specifically including:

[0086] After confirming the starting position and integrity of the data frame, the system enters the deep analysis stage:

[0087] First, the control field is analyzed, which is the key part of understanding the control information of the data frame, including the data direction (upload or download), function code, etc. Then, the system parses the address field to identify the source and destination address of the data frame;

[0088] The system extracts the Information Object Address (IOA), which is a key identifier that uniquely identifies a data item. With the IOA, the system can accurately match the received data with the pre-defined data model to understand the meaning and purpose of the data.

[0089] In the process of parsing the data content, the system not only extracts the actual data value, but also performs a series of error detection measures, including verifying whether the frame length matches the actual received data, checking the error code in the control field, and verifying the integrity and accuracy of the data through means such as CRC (cyclic redundancy check).

[0090] In this embodiment, the target data refers to the specific data content parsed from the data frame.

[0091] In this embodiment, the Kafka message queue is a distributed message queue system with the characteristics of high throughput, strong scalability, low latency, etc. Kafka can process a large number of messages and can reliably store and transmit messages in a distributed environment.

[0092] In this embodiment, the proxy node is a space in the Kafka message queue for storing data, and specifically may be a broker node.

[0093] In this embodiment, the relational database may be a DMDB, which is a space used to store data.

[0094] In this embodiment, the distributed search engine refers to Elasticsearch, which can search and analyze data in a relational database and can be synchronized with the data in the relational database.

[0095] In this embodiment, the data change event refers to the data changes existing in the relational database.

[0096] In this embodiment, hot data refers to specific data corresponding to data change events in relational data.

[0097] In this embodiment, synchronous indexing refers to synchronous retrieval through a distributed search engine.

[0098] The working principle and beneficial effects of the above technical solution are: the master station monitors the TCP connection sent by the slave station, and when a connection exists, the data frame sent by the slave station is received, parsed and verified, thereby ensuring the accuracy and reliability of the data received by the master station; secondly, after the verification is passed, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and then stored in the relational database, thereby ensuring the effective reception and storage of the data sent by the slave station, and improving the reliability of real-time data processing; finally, a distributed search engine is constructed and associated with a relational database, so that when there is a data change event, the hot data corresponding to the data change event is indexed and analyzed in time, thereby improving the real-time processing capability of the data acquisition and processing service; at the same time, it can easily cope with the growth of data volume and the increase of access volume, ensuring the reliability and consistency of the data, and reducing the risk of data loss and errors.

[0099] Embodiment 2:

[0100] Based on Example 1, this example provides a method for processing and receiving real data, such as Figure 2 As shown, in step 1, the master station monitors the TCP connection sent by the slave station in real time, and receives the data frame sent by the slave station when the connection is detected, including:

[0101] Step 101: Obtain an available communication port of the master station, and create a TCP listening socket on the available communication port;

[0102] Step 102: monitoring the TCP connection sent by the slave station based on the creation result, and building a connection socket between the master station and the slave station when the TCP connection is detected;

[0103] Step 103: Based on the construction result, the slave station is controlled to perform protocol encapsulation on the data frame based on the preset protocol, and the protocol encapsulated data frame is sent to the master station based on the connection socket.

[0104] In this embodiment, the available communication port refers to a data communication interface that the master station can currently use.

[0105] In this embodiment, the TCP listening socket is a mechanism for monitoring whether a TCP connection exists.

[0106] In this embodiment, the connection socket is used to connect the master station and the slave station, and is the endpoint of inter-process communication in a computer network. Through the socket, different applications can communicate on the network to implement operations such as sending and receiving data.

[0107] In this embodiment, the preset protocol refers to the 104 protocol.

[0108] The working principle and beneficial effects of the above technical solution are: by creating a TCP listening socket on the available communication port of the master station, it is convenient to monitor the TCP connection sent by the slave station in real time, and it is also convenient to accurately and reliably build a connection socket between the master station and the slave station when a TCP connection exists, which provides a channel and convenience for data transmission. Secondly, after building the connection socket, the data frame of the slave station is encapsulated according to the preset protocol, and the encapsulated data frame is sent to the master station through the connection socket, ensuring the reliability of the master station's reception of data.

[0109] Embodiment 3:

[0110] Based on Example 1, this embodiment provides a method for processing and receiving real data. In step 1, the data frame is parsed and verified based on a preset protocol, including:

[0111] Obtain a preset protocol, extract a format frame in the preset protocol, and parse the data frame based on the format frame to obtain multi-dimensional basic information contained in the data frame;

[0112] Extracting the start character of the identification frame and the standard length of the data frame from the multi-dimensional basic information, and performing length detection on the data frame based on the start character of the identification frame to obtain the actual received length;

[0113] Performing a first check on the data frame based on the standard length and the actual received length of the data frame, and locating the control field of the data frame from the multi-dimensional basic information based on the parsing result of the data frame by the format frame after the first check passes;

[0114] Perform a global traversal of the fields in the control domain, and obtain the target value corresponding to each field based on the global traversal results;

[0115] Determine the function indication of each field according to the field and the target value based on the preset protocol, extract multi-dimensional feature parameters of the data frame based on the function indication, and perform a second check on the extracted multi-dimensional feature parameters based on the function indication;

[0116] After the second check is passed, the information object address of the data frame is extracted from the multi-dimensional basic information based on the parsing result of the format frame on the data frame, and the predefined data model is matched from the preset model library based on the information object address;

[0117] Parse the data frame based on the predefined data model, determine the semantic features and business features corresponding to the data frame, and extract the data value of the data frame;

[0118] Segment the data frame based on a natural language system to obtain multiple target words, and determine the part of speech of each target word based on the natural language system;

[0119] Perform content verification on the data frame based on semantic features, business features and the part of speech of each target time;

[0120] After the content verification is passed, the target data content in the data frame is format converted, and the parsing and verification results of the data frame are recorded based on the format conversion result to obtain a log file.

[0121] In this embodiment, the preset protocol is known in advance.

[0122] In this embodiment, the format frame refers to a data frame with a specific format and structure. It is a data frame formed by encapsulating data according to certain rules and formats, and contains various fields, such as a start flag, address information, control information, data content, check information, an end flag, etc.

[0123] In this embodiment, multi-dimensional basic information refers to the result obtained by parsing and extracting various different information contained in the data frame through the format frame, including specific contents corresponding to the start flag, address information, control information, data content, check information, end flag, etc.

[0124] In this embodiment, the identification frame refers to a data frame that can mark the starting positions of different types of data in the data frame, etc., and is used to distinguish different types of data and identify specific events or operations.

[0125] In this embodiment, the start symbol refers to a symbol in the identification frame used to indicate the starting point of data.

[0126] In this embodiment, the standard length refers to the length of specific data recorded in the data frame, which is known by the slave station when sending.

[0127] In this embodiment, the actual received length refers to the actual data length obtained after the master station receives the data sent by the slave station and performs length detection on the received data.

[0128] In this embodiment, the control field is a key part for understanding the control information of the data frame, including data direction (upload or download), function code, etc.

[0129] In this embodiment, the target value refers to the specific value size corresponding to different fields (such as function code) in the control domain.

[0130] In this embodiment, the function indication refers to the specific meaning of the function code representation determined according to the type of the field and the corresponding value. For example, the function code may represent a data error.

[0131] In this embodiment, multi-dimensional feature parameter extraction refers to extracting corresponding feature parameters from a data frame according to a function indication, thereby facilitating corresponding matching verification of the received data according to the function indication of the function code. For example, when the function indication is the type or structure of the data, the extracted feature parameters are used to verify whether the current data is of the type or structure specified by the function indication.

[0132] In this embodiment, the information object address refers to the specific source address information corresponding to the data frame.

[0133] In this embodiment, the preset model library is constructed in advance and contains a variety of different data models.

[0134] In this embodiment, the predefined data model refers to a model that is retrieved from a preset model library according to the information object address and is capable of determining the current data content.

[0135] In this embodiment, the semantic feature refers to the specific meaning corresponding to the data frame obtained after the data frame is parsed accordingly through a predefined data model.

[0136] In this embodiment, the service feature refers to the service type that can be represented by the data contained in the data frame or the service purpose that can be achieved.

[0137] In this embodiment, the natural language system is used to represent the word structure between data and the part of speech of each word, wherein the part of speech can be the subject or object in a sentence.

[0138] In this embodiment, format conversion refers to converting data from original binary or hexadecimal format into a format recognizable by the system (such as floating point numbers, integers, character strings, etc.).

[0139] The working principle and beneficial effects of the above technical solution are: by parsing the data frame according to the format frame in the preset protocol, the multi-dimensional basic information contained in the data frame can be accurately and effectively determined. At the same time, the multi-dimensional basic information is parsed to strictly verify the length of the data frame, the integrity and accuracy of the data, thereby ensuring the comprehensiveness and reliability of the data verification. Secondly, after the verification is passed, the information object address corresponding to the data frame is obtained, and the corresponding predefined data model is retrieved according to the information object address, so that the data frame is parsed through the predefined data model, and the semantic features and business features corresponding to the data frame are effectively determined. Finally, the data frame is segmented according to the natural language system, and the content of the data frame is effectively verified according to the segmentation results and the obtained semantic features and business features. Finally, the format conversion is performed after the verification is passed, and the corresponding log file is generated at the same time, ensuring the accuracy and reliability of the data received by the main station.

[0140] Embodiment 4:

[0141] Based on Example 1, this embodiment provides a method for processing and receiving real data. In step 1, the data frame is parsed and verified based on a preset protocol, including:

[0142] Obtain the parsing and verification links of the data frame, and determine the corresponding processing plan when each parsing and verification link fails based on management requirements;

[0143] A multi-touch jump mechanism is constructed based on the analysis and verification links, and the corresponding processing scheme when each analysis and verification link fails is associated and bound with the multi-touch jump mechanism;

[0144] Based on the association binding result, the result status of each analysis and verification link is received in real time, and when the real-time received result is judged as not passed, a burst response is made to the current analysis and verification link based on the multi-touch jump mechanism;

[0145] Based on the emergency response results, emergency processing is performed on the current analysis and verification links according to the processing plan.

[0146] In this embodiment, the management requirements are known in advance and are used to characterize the specific processing measures corresponding to when the parsing and verification steps fail.

[0147] In this embodiment, the processing scheme is pre-set. For example, if any error is found, the system will immediately trigger the error handling mechanism, which may include requesting the slave station to resend the data frame, recording a detailed error log, or disconnecting the slave station when necessary.

[0148] In this embodiment, the multi-touch point jump mechanism is formulated according to the analysis and verification links, and is used to immediately stop subsequent analysis and verification links when there is a failure in the analysis and verification links.

[0149] The working principle and beneficial effects of the above technical solution are: by determining the parsing and verification links of the data frame, and locking the processing plan for each parsing and verification link when it fails, data support is provided for burst response; secondly, a multi-contact jump mechanism is constructed according to the parsing and verification links, and the multi-contact jump mechanism is associated and bound with the corresponding processing plan for each parsing and verification link when it fails, and finally a reliable burst response is achieved for each parsing and verification link according to the result status of each parsing and verification link, thereby improving the reliability of data processing and reception.

[0150] Embodiment 5:

[0151] Based on Example 1, this embodiment provides a method for processing and receiving real data. In step 2, after verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is stored in the relational database in turn based on the push result, including:

[0152] Obtain the structural distribution characteristics of each proxy node in the Kafka message queue, and determine the storage path between each proxy node in the Kafka message queue for the data to be stored based on the structural distribution characteristics;

[0153] At the same time, extracting basic attributes of each proxy node, and determining a rated operating state of each proxy node based on the basic attributes, and determining a resource allocation amount for each proxy node based on the rated operating state;

[0154] Configure the Kafka message queue based on the storage path and resource allocation, and extract the data topic of each proxy node in the Kafka message queue based on the configuration result. At the same time, convert the parsed target data into key-value pairs based on the operation requirements of the Kafka message queue;

[0155] Match the key in the key-value pair with the data subject of each proxy node, and determine the proxy node corresponding to different target data based on the matching result;

[0156] Based on the storage path, the values ​​in the key-value pair are pushed to the corresponding proxy nodes in the Kafka message queue in real time, and the data sequence of each proxy node is obtained;

[0157] Obtain the API interface of the relational database, and store the data sequence of each proxy node in the Kafka message queue to the relational database based on the API interface.

[0158] In this embodiment, the structural distribution feature refers to the distribution of each proxy node in the Kafka message queue, so as to facilitate determining the storage path for the data to be stored according to the structural distribution feature.

[0159] In this embodiment, the basic attributes refer to information such as the type of data that can be stored in each proxy node and the maximum amount of data that can be stored during operation.

[0160] In this embodiment, the rated operating state refers to the operating state corresponding to each proxy node under full load.

[0161] In this embodiment, the resource allocation amount refers to the amount of resources that each proxy node needs to be provided by the system during normal operation, which is used to ensure the effective operation of each proxy node.

[0162] In this embodiment, the data subject refers to the type of data that each proxy node can store.

[0163] In this embodiment, the key-value pair refers to a data format that can be recognized by each proxy node, wherein the key plays a key role in determining to which partition the data will be sent, and the value carries the actual data content.

[0164] In this embodiment, the data sequence refers to the result obtained after storing different to-be-stored data in the proxy node in sequence, and is obtained based on a time series.

[0165] The working principle and beneficial effects of the above technical solution are: by determining the structural distribution characteristics of each proxy node in the Kafka message queue, the storage path of the data to be stored between the proxy nodes in the Kafka message queue can be effectively determined according to the structural distribution characteristics. At the same time, the basic properties of each proxy node are analyzed to effectively determine the resource allocation required for the operation of each proxy node according to the analysis results. Secondly, the Kafka message queue is configured with the storage path and resource allocation, so that the data to be stored can be effectively stored in the corresponding proxy node according to the configuration results. Finally, the data sequence in the proxy node is synchronously sent to the relational database for storage, ensuring the effective reception and storage of the data sent by the slave station, and improving the reliability of real-time data processing.

[0166] Embodiment 6:

[0167] Based on Example 5, this embodiment provides a method for processing and receiving real-time data, which pushes the values ​​in the key-value pairs to the corresponding proxy nodes in the Kafka message queue in real time based on the storage path, including:

[0168] Obtain target push content based on the value in the key-value pair, and split the target push content into N content blocks;

[0169] Traverse the characters in each content block in turn, obtain redundant information in each content block, and remove the redundant information;

[0170] Compress the content blocks after redundant information is removed based on a preset compression method, and push the compressed content blocks to the corresponding proxy nodes in the Kafka message queue in real time;

[0171] The storage space status of each proxy node is periodically monitored based on the push results, and when the storage space status of the proxy node meets the preset cleaning conditions, the deletion data of the stored data in each proxy node is determined based on the time log, and the deleted data is eliminated.

[0172] In this embodiment, the target push content refers to the specific data represented by the value in the key-value pair.

[0173] In this embodiment, the content block refers to the result obtained by splitting the target pushed content, and is a part of the original target pushed content.

[0174] In this embodiment, redundant information refers to repeated data in each content block.

[0175] In this embodiment, the preset compression method is set in advance, such as GZIP, Snappy, etc.

[0176] In this embodiment, the storage space status refers to the change of the storage space in each proxy node.

[0177] In this embodiment, the preset cleaning condition is set in advance. For example, when the remaining storage space is less than a preset threshold, the cleaning condition is met.

[0178] In this embodiment, the time log is used to record the storage duration of different data in the proxy node and the specific time information corresponding to the storage.

[0179] The working principle and beneficial effects of the above technical solution are: by verifying the target push content, redundant information in the target push content is eliminated, thereby ensuring the reliability of the target push content; secondly, the processed target push content is compressed, and the target push content is effectively pushed to the corresponding proxy node according to the compression result; finally, the storage space status of each proxy node is periodically monitored, and when the storage space status meets the preset cleaning conditions, the storage space of each proxy node is released, thereby ensuring the reliability of data reception.

[0180] Embodiment 7:

[0181] Based on Example 1, this embodiment provides a method for processing and receiving real-time data. In step 3, a distributed search engine is constructed and associated with a relational database, including:

[0182] Obtaining a framework structure of a relational database, and determining a node layout when searching for data in the relational database based on the framework result, and determining a communication address of each node in the relational database based on the node layout;

[0183] Obtain the index source of each node based on the communication address, and build the local index of each node based on the index source;

[0184] Adding the local index to the script file of the target search engine for the first time, and obtaining a distributed search engine based on the result of the first addition;

[0185] The distributed search engine is interfaced with the relational database, and an identifier of the relational database is obtained based on the interface connection result;

[0186] The identifier is added to the script file of the target search engine for a second time, and the association between the distributed search engine and the relational database is completed based on the second addition.

[0187] In this embodiment, the framework structure refers to the distribution of different data layers in a relational database, so as to facilitate the data nodes where different data are located when searching for data.

[0188] In this embodiment, the node layout represents the specific storage locations corresponding to different data in the relational database.

[0189] In this embodiment, the index source refers to the specific communication address corresponding to each node.

[0190] In this embodiment, the local index refers to a label or basis for locating each node.

[0191] In this embodiment, the target search engine is constructed in advance.

[0192] In this embodiment, the script file refers to the software program that the target search engine relies on when running.

[0193] The working principle and beneficial effects of the above technical solution are: by determining the framework structure of the relational database, the node layout during data search is effectively determined according to the framework structure, thereby determining the local index of each node according to the node layout, and secondly, configuring the parameters of the script file of the target search engine according to the local index. At the same time, the identifier of the relational database is added to the script file of the target search engine, so as to realize the reliable association between the distributed search engine and the relational database, which provides great convenience for data search and access.

[0194] Embodiment 8:

[0195] Based on Example 1, this embodiment provides a method for processing and receiving real-time data. In step 4, data change events in a relational database are monitored in real time based on the association results, and hot data corresponding to the data change events are synchronously indexed and analyzed according to a distributed search engine based on the monitoring results, including:

[0196] Monitor the status of stored data in the relational database in real time, and start the distributed search engine synchronously when data change events are detected;

[0197] Based on the synchronization start result, the distributed search engine is controlled to search for the data change node, and the hot data corresponding to the data change event is determined based on the search result;

[0198] Hot data is indexed based on a distributed search engine, and the indexed hot data is analyzed and processed.

[0199] In this embodiment, the data change node refers to a specific data storage node corresponding to the data change event in the relational database.

[0200] In this embodiment, indexing hot data based on a distributed search engine refers to accessing and retrieving hot data corresponding to a data change event through a distributed search engine.

[0201] The working principle and beneficial effects of the above technical solution are: by performing real-time status monitoring on the stored data in the relational database, it is convenient to synchronize the startup of the distributed search engine in time when there is a data change event, and to lock the hot data corresponding to the data change event according to the synchronization startup result. Finally, the hot data is indexed and processed by the distributed search engine, which improves the reliability of data reception and processing, and improves the real-time processing capability of data acquisition and processing services.

[0202] Embodiment 9:

[0203] This embodiment provides a system for processing and receiving real data, such as Figure 3As shown, including:

[0204] The data receiving and parsing module is used to monitor the TCP connection sent by the slave station in real time based on the master station, and when the connection is detected, receive the data frame sent by the slave station, and parse and verify the data frame based on the preset protocol;

[0205] The data storage module is used to push the parsed target data to the corresponding proxy node in the Kafka message queue in real time after verification, and store the target data in each proxy node in the relational database in turn based on the push results;

[0206] The association module is used to build a distributed search engine and associate the distributed search engine with a relational database;

[0207] The monitoring management module is used to monitor data change events in the relational database in real time based on the correlation results, and synchronously index and analyze the hot data corresponding to the data change events according to the distributed search engine based on the monitoring results.

[0208] The working principle and beneficial effects of the above technical solution are: the master station monitors the TCP connection sent by the slave station, and when a connection exists, the data frame sent by the slave station is received, parsed and verified, thereby ensuring the accuracy and reliability of the data received by the master station; secondly, after the verification is passed, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and then stored in the relational database, thereby ensuring the effective reception and storage of the data sent by the slave station, and improving the reliability of real-time data processing; finally, a distributed search engine is constructed and associated with a relational database, so that when there is a data change event, the hot data corresponding to the data change event is indexed and analyzed in time, thereby improving the real-time processing capability of the data acquisition and processing service; at the same time, it can easily cope with the growth of data volume and the increase of access volume, ensuring the reliability and consistency of the data, and reducing the risk of data loss and errors.

[0209] Embodiment 10:

[0210] Based on Example 9, this embodiment provides a system for processing and receiving real data, and a data receiving and parsing module, including:

[0211] A monitoring mechanism creation unit, used for obtaining an available communication port of the master station and creating a TCP monitoring socket on the available communication port;

[0212] A connection unit, for monitoring the TCP connection sent by the slave station based on the creation result, and building a connection socket between the master station and the slave station when the TCP connection is detected;

[0213] The data receiving unit is used to control the slave station to perform protocol encapsulation on the data frame based on the preset protocol based on the construction result, and send the protocol encapsulated data frame to the master station based on the connection socket.

[0214] The working principle and beneficial effects of the above technical solution are: by creating a TCP listening socket on the available communication port of the master station, it is convenient to monitor the TCP connection sent by the slave station in real time, and it is also convenient to accurately and reliably build a connection socket between the master station and the slave station when a TCP connection exists, which provides a channel and convenience for data transmission. Secondly, after building the connection socket, the data frame of the slave station is encapsulated according to the preset protocol, and the encapsulated data frame is sent to the master station through the connection socket, ensuring the reliability of the master station's reception of data.

[0215] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for processing and receiving real data, characterized in that: include: Step 1: Based on the master station, the TCP connection sent by the slave station is monitored in real time, and when the connection is detected, the data frame sent by the slave station is received, and the data frame is parsed and verified based on the preset protocol; Step 2: After verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is stored in the relational database in turn based on the push results; Step 3: Build a distributed search engine and associate the distributed search engine with a relational database; Step 4: Based on the correlation results, the data change events in the relational database are monitored in real time, and based on the monitoring results, the hot data corresponding to the data change events are synchronously indexed and analyzed using a distributed search engine.

2. A method for processing and receiving real data according to claim 1, characterized in that: In step 1, the master station monitors the TCP connection sent by the slave station in real time, and receives the data frame sent by the slave station when the connection is detected, including: Get the available communication port of the master station and create a TCP listening socket on the available communication port; Based on the creation result, the TCP connection sent by the slave is monitored, and when the TCP connection is detected, a connection socket between the master and the slave is established; Based on the construction result, the slave station is controlled to perform protocol encapsulation on the data frame based on the preset protocol, and the protocol encapsulated data frame is sent to the master station based on the connection socket.

3. A method for processing and receiving real data according to claim 1, characterized in that: In step 1, the data frame is parsed and verified based on the preset protocol, including: Obtain a preset protocol, extract a format frame in the preset protocol, and parse the data frame based on the format frame to obtain multi-dimensional basic information contained in the data frame; Extracting the start character of the identification frame and the standard length of the data frame from the multi-dimensional basic information, and performing length detection on the data frame based on the start character of the identification frame to obtain the actual received length; Performing a first check on the data frame based on the standard length and the actual received length of the data frame, and locating the control field of the data frame from the multi-dimensional basic information based on the parsing result of the data frame by the format frame after the first check passes; Perform a global traversal of the fields in the control domain, and obtain the target value corresponding to each field based on the global traversal results; Determine the function indication of each field according to the field and the target value based on the preset protocol, extract multi-dimensional feature parameters of the data frame based on the function indication, and perform a second check on the extracted multi-dimensional feature parameters based on the function indication; After the second check is passed, the information object address of the data frame is extracted from the multi-dimensional basic information based on the parsing result of the format frame on the data frame, and the predefined data model is matched from the preset model library based on the information object address; Parse the data frame based on the predefined data model, determine the semantic features and business features corresponding to the data frame, and extract the data value of the data frame; Segment the data frame based on a natural language system to obtain multiple target words, and determine the part of speech of each target word based on the natural language system; Perform content verification on the data frame based on semantic features, business features and the part of speech of each target time; After the content verification is passed, the target data content in the data frame is format converted, and the parsing and verification results of the data frame are recorded based on the format conversion result to obtain a log file.

4. A method for processing and receiving real data according to claim 1, characterized in that: In step 1, the data frame is parsed and verified based on the preset protocol, including: Obtain the parsing and verification links of the data frame, and determine the corresponding processing plan when each parsing and verification link fails based on management requirements; A multi-touch jump mechanism is constructed based on the analysis and verification links, and the corresponding processing scheme when each analysis and verification link fails is associated and bound with the multi-touch jump mechanism; Based on the association binding result, the result status of each analysis and verification link is received in real time, and when the real-time received result is judged as not passed, a burst response is made to the current analysis and verification link based on the multi-touch jump mechanism; Based on the emergency response results, emergency processing is performed on the current analysis and verification links according to the processing plan.

5. A method for processing and receiving real data according to claim 1, characterized in that: In step 2, after verification, the parsed target data is pushed to the corresponding proxy node in the Kafka message queue in real time, and the target data in each proxy node is stored in the relational database in turn based on the push result, including: Obtain the structural distribution characteristics of each proxy node in the Kafka message queue, and determine the storage path between each proxy node in the Kafka message queue for the data to be stored based on the structural distribution characteristics; At the same time, extracting basic attributes of each proxy node, and determining a rated operating state of each proxy node based on the basic attributes, and determining a resource allocation amount for each proxy node based on the rated operating state; Configure the Kafka message queue based on the storage path and resource allocation, and extract the data topic of each proxy node in the Kafka message queue based on the configuration result. At the same time, convert the parsed target data into key-value pairs based on the operation requirements of the Kafka message queue; Match the key in the key-value pair with the data subject of each proxy node, and determine the proxy node corresponding to different target data based on the matching result; Based on the storage path, the values ​​in the key-value pair are pushed to the corresponding proxy nodes in the Kafka message queue in real time, and the data sequence of each proxy node is obtained; Obtain the API interface of the relational database, and store the data sequence of each proxy node in the Kafka message queue to the relational database based on the API interface.

6. A method for processing and receiving real data according to claim 5, characterized in that: Based on the storage path, the values ​​in the key-value pair are pushed to the corresponding proxy nodes in the Kafka message queue in real time, including: Obtain target push content based on the value in the key-value pair, and split the target push content into N content blocks; Traverse the characters in each content block in turn, obtain redundant information in each content block, and remove the redundant information; Compress the content blocks after redundant information is removed based on a preset compression method, and push the compressed content blocks to the corresponding proxy nodes in the Kafka message queue in real time; The storage space status of each proxy node is periodically monitored based on the push results, and when the storage space status of the proxy node meets the preset cleaning conditions, the deletion data of the stored data in each proxy node is determined based on the time log, and the deleted data is eliminated.

7. A method for processing and receiving real data according to claim 1, characterized in that: In step 3, a distributed search engine is constructed and associated with a relational database, including: Obtaining a framework structure of a relational database, and determining a node layout when searching for data in the relational database based on the framework result, and determining a communication address of each node in the relational database based on the node layout; Obtain the index source of each node based on the communication address, and build the local index of each node based on the index source; Adding the local index to the script file of the target search engine for the first time, and obtaining a distributed search engine based on the result of the first addition; The distributed search engine is interfaced with the relational database, and an identifier of the relational database is obtained based on the interface connection result; The identifier is added to the script file of the target search engine for a second time, and the association between the distributed search engine and the relational database is completed based on the second addition.

8. A method for processing and receiving real data according to claim 1, characterized in that: In step 4, data change events in the relational database are monitored in real time based on the association results, and hot data corresponding to the data change events are synchronously indexed and analyzed using a distributed search engine based on the monitoring results, including: Monitor the status of stored data in the relational database in real time, and start the distributed search engine synchronously when data change events are detected; Based on the synchronization start result, the distributed search engine is controlled to search for the data change node, and the hot data corresponding to the data change event is determined based on the search result; Hot data is indexed based on a distributed search engine, and the indexed hot data is analyzed and processed.

9. A system for processing and receiving real data, characterized in that: include: The data receiving and parsing module is used to monitor the TCP connection sent by the slave station in real time based on the master station, and when the connection is detected, receive the data frame sent by the slave station, and parse and verify the data frame based on the preset protocol; The data storage module is used to push the parsed target data to the corresponding proxy node in the Kafka message queue in real time after verification, and store the target data in each proxy node in the relational database in turn based on the push results; The association module is used to build a distributed search engine and associate the distributed search engine with a relational database; The monitoring management module is used to monitor data change events in the relational database in real time based on the correlation results, and synchronously index and analyze the hot data corresponding to the data change events according to the distributed search engine based on the monitoring results.

10. A system for processing and receiving real data according to claim 9, characterized in that: Data receiving and parsing module, including: A monitoring mechanism creation unit is used to obtain an available communication port of the master station and create a TCP monitoring socket on the available communication port; A connection unit, for monitoring the TCP connection sent by the slave station based on the creation result, and building a connection socket between the master station and the slave station when the TCP connection is detected; The data receiving unit is used to control the slave station to perform protocol encapsulation on the data frame based on the preset protocol based on the construction result, and send the protocol encapsulated data frame to the master station based on the connection socket.

Images