A multi-link data reception fusion method based on multi-threading

Abstract

The present invention relates to a multi-link data receiving fusion method based on multithreading, and belongs to the field of high-availability data fusion. The present invention is applied to remote data transmission and storage with high reliability, node service cluster construction, and has high reliability. It can be applied to application environments such as GPS or Beidou positioning objects and object tracking display services, so that the data provider does not need to worry about the data transmission termination caused by problems such as server shutdown and power failure, or node application process stopping operation. When a power outage occurs in one of the service clusters of multiple computer rooms, users do not have to worry about data transmission interruption, thereby improving the stability of data transmission, reducing the operation and maintenance costs of supervisors, and allowing users to only care about data display services. At the same time, the application has configurable data source connection, configurable data protocol analysis, and configurable fusion filtering algorithm. Users can customize data source connection, customize data protocol and data filtering method for parsing, filtering and forwarding data.

Description

Technical Field

[0001] The present invention belongs to the field of high-availability data fusion, and in particular relates to a multi-link data receiving and fusion method based on multi-threading. Background Art

[0002] As Beidou positioning technology becomes more mature and widely used, more and more mobile phones, microcomputers, and web pages provide users with convenient and fast user positioning experience services. The content of the Beidou data protocol is more diverse, with both digital forms of longitude and latitude and time and date, as well as string forms such as daily text. However, traditional positioning technology cannot solve the positioning accuracy of data, the long data time transmission cycle, and the lack of solutions for sustainable data transmission. Therefore, it is no longer suitable for positioning services that are used more and more frequently and require high reliability.

[0003] In order to solve many problems existing in the application process of Beidou positioning technology, such as server shutdown, disappearance of server application process, data loss, data timing confusion, Beidou positioning data offset, etc., and to make the data transmission more stable and efficient in the process of combining Beidou positioning technology with applications, the present invention aims to apply multi-threading technology and fusion filtering algorithm technology to the data source connection, data protocol parsing and error data filtering of Beidou positioning, so as to solve various problems arising in the process of data receiving and forwarding.

[0004] The present invention is applied to a data transmission service of Beidou / GPS positioning data, mainly solving the problem of sustainable transmission of basic data with multi-link data filtering fusion, generalization of multiple protocol analysis, and generalization of multiple connections. The specific problems to be solved include:

[0005] To solve the problem of data loss when downtime occurs and application processes disappear, it is necessary to build a node cluster, which increases stability and reduces operation and maintenance costs.

[0006] To solve the problems of processing too many types of protocols and processing new protocols in uncertain formats, it is necessary to design a set of universal configuration files for parsing multiple data protocols. Through the configuration instructions of the configuration files, users can customize the data protocol format, reduce user operation steps and reduce R&D costs.

[0007] It solves the problems of data redundancy and sending erroneous data. According to the characteristics of user-defined data, users can customize data filtering algorithms to filter out erroneous data.

[0008] Solved the problem of complex and diverse data sources.

[0009] Solve the status synchronization problem of node cluster construction.

[0010] Node recovery startup status reply problem.

[0011] Solve the problem of request sending failure due to single or multiple server shutdown, process loss, etc. during the synchronization of all node status. Summary of the invention

[0012] 1. Technical issues to be resolved

[0013] The technical problem to be solved by the present invention is how to provide a multi-link data receiving fusion method based on multi-threading to solve many problems existing in the application process of Beidou positioning technology, such as server shutdown, disappearance of server application process, data loss, data timing confusion, Beidou positioning data offset, etc.

[0014] (II) Technical solution

[0015] In order to solve the above technical problems, the present invention proposes a multi-link data receiving fusion method based on multi-threading. The method is based on a fusion system, which is composed of multiple nodes to form a cluster. Each node is connected to three targets. The first target is a data source, which is used to receive data. The second target is a database, which is a relational database and is used to record the cluster status. The third target is a cache middleware Redis, which is used to execute a data fusion script protocol. The method includes:

[0016] In each node, the receiver form, data protocol form and data filtering algorithm are defined through a custom configuration file, and multiple threads are established through thread pool technology. Each thread establishes a receiver and runs data receiving and data processing services at the same time. The data processing services of each receiver will not affect each other. After parsing, the data from multiple data sources are aggregated to perform data deduplication and data filtering operations.

[0017] Each node is an independently running node, and each node uses multi-threading technology to connect to multiple data sources and run in parallel. Each node deduplicates the data received by the data source, and then calculates the data distance and reception time difference between the two nearest points to determine whether it is erroneous data. If so, it is filtered and finally the processed data is sent to the cache middleware Redis. The script technology and SET collection of the cache middleware Redis are used to perform data deduplication and data transmission, and the cached outdated data can be destroyed at a scheduled time.

[0018] The cache middleware Redis is set outside the application cluster. Each node in the cluster connects to the cache middleware Redis through the TCP protocol. The data sent is large enough to be sent to the SET set of the cache middleware Redis. If the SET set does not exist, the data is sent to the blocking queue. The cache middleware Redis supports connections in multiple languages ​​and can consume blocking queue data.

[0019] Each node is deployed on a different server. Each node is a process in the server. The cluster is built on multiple servers. Multiple servers can be distributed in multiple computer rooms. If a server goes down or restarts, the process disappears and the cluster loses a node. However, other nodes are still running and data will not be lost. Because all nodes receive the same data, the sent data is also the same. Even if a node server fails, it does not affect other nodes from receiving and sending data. Due to the cache middleware Redis script technology and the non-repetitive nature of SET sets, the data of all nodes is sent to this SET set to determine whether it exists. If it exists, it means that it has been received. If it does not exist, it means that the data has not been received. Therefore, when a node hangs up, the data can be resumed within 1 second, and the data is unique.

[0020] The data received by each node has data time. First, each node will filter out the data with disordered time. However, the time sent by each node to the cache middleware Redis is different. Therefore, there are two solutions to customize the script protocol of the cache middleware Redis. The first solution is that because the data of each node is ordered, you only need to set the SET collection in the cache middleware Redis, send the data of all nodes to the SET collection, and use the non-repeatability of the SET collection to determine whether the data is received. If there is no data in the SET collection, cache the data to the SET collection and send the data to the blocking queue. Because the queue has the characteristics of first-in-first-out, the final consumer of the data has order; the second solution is to set the SET collection in the cache middleware Redis. The permission Key is set in s, and has a validity period of 5 seconds. This 5-second validity period can be customized to any positive integer value. The value of the Key is the IP address of the node. The allocation rule of this permission is that the node that sends data first has this permission, that is, the value of the Key is the IP address of the node that sends data first, then this node will have 5 seconds to send data to the SET collection, and then determine whether the collection receives the same data. If not, cache the data to the SET collection and send the data to the blocking queue; after 5 seconds, create an empty Key again. If any node sends data first, the IP address of this node will be assigned to this Key. Since the data of each node is time-series, the data finally collected in the queue is time-series;

[0021] The user sends a request to node 1. Node 1 obtains the cluster configuration through the configuration file, asynchronously forwards the request to other nodes, and synchronizes the status so that all nodes in the cluster have the same receiving and forwarding status at the same time. When the synchronization is successful, the current status information is saved in the database. When a node crashes or the request is unavailable, the node can be removed without affecting the request operations of other nodes. If it is not a crash or the request is unavailable, when a node service request fails, in order to make all nodes consistent, the previous state of the node stored in the database is used to read the previous state of the database and restore the previous state of the node. The database is used to store node status, connection and forwarding information. When the node hangs up, it can be restarted to restore the connection status and other states.

[0022] (III) Beneficial effects

[0023] The present invention proposes a multi-link data receiving and fusion method based on multi-threading. The present invention discloses a data fusion and filtering method for multi-threaded and multi-link data. The main advantages are embodied in the following aspects:

[0024] 1. We have designed a multi-link data fusion solution. Even if the server goes down or the application process disappears, data can be restored within one second, so that customers do not have to worry about power outages. There is also a startup script, which allows you to run the application by turning on the server without too much operation.

[0025] 2. A cluster building solution is designed. Using multi-threading technology, a request is sent to a node, and this node will asynchronously send the same request to all other nodes through multi-threading. In this way, the problems of cluster configuration file upload, configuration file synchronization, and sending status synchronization can be solved. However, when a server crashes, the application node on this server will disappear, and other nodes will remove this node without affecting the normal operation of other nodes. However, when a node has a request problem, the operation of rolling back the status of all nodes can also be set to make the status of all nodes consistent, thereby reducing operation and maintenance operations.

[0026] 3. A method that supports custom data filtering algorithms is designed. Users can customize what kind of data to obtain according to their own data needs, and customize their own filtering methods by uploading binary algorithm files and configuration files, which is simple and effective.

[0027] 4. A simple service building method that supports connecting to multiple data sources is designed. Using multi-threading technology, one node can connect to multiple data sources. By configuring the basic information of multiple data sources, forwarding destinations, routing tables and other writing operations, and uploading configuration files, the application nodes required by users can be customized and saved in the database. When the node is restarted, the last configured connection information is still used, and there is no need to upload the configuration file again.

[0028] 5. A universal solution for data protocol parsing is designed. By customizing the configuration file protocol, the function of application nodes to process data protocols is universal, reducing the developers' research and development of new protocols. In addition, only one configuration file needs to be uploaded to build an environment for custom data protocol parsing algorithms, thereby reducing the number of user-defined data protocol operations.

[0029] 6. Designed an overall architecture, including cache middleware Redis, database and node application, and supports clients written in multiple languages ​​to access and obtain data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a system architecture diagram of the present invention;

[0031] Figure 2 For business process diagram;

[0032] Figure 3 It is a multi-link deployment diagram;

[0033] Figure 4 It is the node cluster status synchronization diagram;

[0034] Figure 5 It is the difference data processing flow;

[0035] Figure 6 This is the flow chart of the difference algorithm. DETAILED DESCRIPTION

[0036] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0037] The present invention belongs to the field of high-availability data fusion, and is a multi-link data receiving and fusion method based on multi-threading, specifically involving multi-threading technology, filtering algorithm technology, and data protocol analysis technology.

[0038] The present invention provides a solution for data filtering fusion, node state synchronization and node state recovery, and combines Beidou technology with multi-threading technology. The present invention provides a multi-link data receiving fusion filtering method based on multi-threading.

[0039] The method of the present invention includes application to remote data transmission and storage with high reliability, node service cluster construction, and can be applied to application environments such as GPS or Beidou positioning objects and object tracking display services, so that the data provider does not need to worry about the server shutdown and power failure, or the node application process stopping running and other problems causing the data transmission to be terminated. When a power outage occurs in one of the service clusters of multiple computer rooms, the user does not have to worry about the interruption of data transmission, thereby improving the stability of data transmission, reducing the operation and maintenance costs of supervisors, and allowing users to only care about data display services. At the same time, the application has configuration functions such as configurable data source connection, configurable data protocol analysis, and configurable fusion filtering algorithm. Users can customize data source connections, customize data protocols and data filtering methods for parsing, filtering and forwarding data.

[0040] A multi-link data receiving and fusion method based on multi-threading. The present invention can start multiple nodes as a cluster. Each node has data transmission and reception and data fusion functions, so there will be no data duplication or data loss. The specific architecture is as follows Figure 1 shown.

[0041] The method of the present invention is based on a fusion system, which is composed of multiple nodes to form a cluster. Each node is connected to three targets. The first target is a data source, which can have multiple data sources and is used to receive data. The second target is a database, which is a relational database, such as Oracle, MySQL, or domestic databases such as DAMO and SINTONG, etc., which is used to record the cluster status. The third target is a cache middleware Redis, which is used to execute a data fusion script protocol. The scheme includes the following problem-solving solutions.

[0042] 1.1 Multi-threaded processing of data from multiple data sources

[0043] In each node, the receiver form, data protocol form and data filtering algorithm are defined through a custom configuration file, and multiple threads are established through thread pool technology. Each thread establishes a receiver and runs data reception and data processing services at the same time. The data processing services of each receiver will not affect each other. After parsing, the data from multiple data sources are aggregated to perform data deduplication and data filtering operations.

[0044] 1.2 Multi-link Data Fusion

[0045] like Figure 1As shown in the figure, the cluster consists of multiple nodes, each of which is an independently running node, and each node connects to multiple data sources through multi-threading technology and runs in parallel. However, since the data received by each node contains duplicate data and erroneous data, in order to ensure that the data sent by the cluster is not redundant and does not generate errors, each node first deduplicates the data received by the data source, and then calculates the data distance and reception time difference between the two nearest points to determine whether it is erroneous data. If so, it is filtered, and finally the processed data is sent to the cache middleware Redis. Relying on the script technology and SET collection of the cache middleware Redis, data deduplication and data transmission are performed, and outdated data is destroyed regularly. Relying on the high efficiency of the script technology, data recovery and transmission can be achieved within 1 second. The specific script protocol is as follows Figure 2 shown.

[0046] (1) The present invention provides data to the outside world and can support different programming languages.

[0047] The cache middleware Redis is set outside the application cluster. As long as the network is connected, it can be in different computer rooms or even different regions. The cache middleware Redis is a cluster. Each node in the cluster connects to the cache middleware Redis through the TCP protocol and writes data to the SET collection and blocking queue of the cache middleware Redis. The cache middleware Redis supports multiple language connections, so it can provide consumption data of systems developed in different development languages.

[0048] (2) The present invention has high availability and the fused data is unique

[0049] Each node is deployed on a different server. Each node is a process in the server. The cluster is built on multiple servers. Multiple servers can be distributed in multiple computer rooms. If a server goes down or the server is restarted, the process disappears and the cluster loses a node. However, other nodes are still running and data will not be lost. Figure 3 As shown. All nodes receive the same data, so the data sent is also the same. Even if a node's server fails, it does not affect other nodes from receiving and sending data. Due to the cache middleware Redis script technology and the non-repetitive nature of the SET set, all nodes' data are sent to this SET set to determine whether it exists. If it exists, it means it has been received. If it does not exist, it means the data has not been received. Therefore, when a node fails, data can be resumed within 1 second, and the data is unique. Then the system that needs to display the object's motion status receives the data and displays the object's running status on the map.

[0050] (3) The fused data is orderly

[0051] The data received by each node has data time, such as Figure 2 As shown, first, each node will filter out the data with disordered time order. However, the time that each node sends to the cache middleware Redis is different. Therefore, there are two solutions to customize the script protocol of the cache middleware Redis. The first solution is that because the data of each node is orderly, you only need to set the SET set in the cache middleware Redis, use the non-repetitiveness of the SET set, send the data to the SET set, and judge whether the data is received. If there is no data in the SET set, cache the data to the SET set, and then send the data to the blocking queue. Because the queue has the first-in-first-out feature, the final consumer data has order. The second solution is to set the permission Key in the cache middleware Redis, and it has a time limit of 5 seconds. This 5-second time limit can be customized to any positive integer value. The value of the Key is the IP address of the node. The allocation rule of this permission is that the node that sends the data first has this permission, that is, the value of this Key is the IP address of the node that sends the data first, then this node will have 5 seconds to send data to the SET set, and then judge whether the set receives the same data. If not, cache the data to the SET set, and then send the data to the blocking queue. After 5 seconds, an empty Key will be created again. If any node sends data first, the IP address of this node will be assigned to this Key. Since the data of each node is time-series, the data finally collected in the queue is also time-series.

[0052] 1.3 Cluster Setup

[0053] like Figure 3 As shown, a cluster is composed of one or more nodes. Each node has the same data receiving and data sending services. Each node is connected to the same data source. There may be multiple data sources, so multiple data sources send the same data to each node at the same time. Just configure the IP addresses and ports of all nodes in the application configuration file, use server sockets to write rules and connect with commas, and write node cluster configurations, such as 127.0.0.1:8080, 127.0.0.1:8081, 127.0.0.1:8082

[0054] 1.4 Universal analysis solution for data protocols

[0055] Through the configuration file data protocol format configuration, configuration protocol name, protocol ID, protocol byte composition, such as the first byte is the word connector, the second byte is the data type, the third byte is the data quantity, the fourth to seventh bytes are the longitude value of the data, etc., and the programming language reflection technology is used to configure the protocol data in the configuration file for each byte or each byte array parsing method name, and the reflection mechanism and mapping mechanism of the programming language are used to map the name to the executable method of the byte data, and the data is parsed through these methods. Users also upload their own method files to the database and load them into the memory, and then use the reflection technology to obtain the executable method, and finally cache the method name and method in the application through the mapping relationship. When the data needs to be parsed, it is only necessary to obtain the parsing method through the configuration file protocol parsing data method name, and the application can parse the data through these methods.

[0056] 1.5 Node connection status setting

[0057] The database is a relational database, such as Oracle, MySQL, or domestic databases such as DAMO and SENTONG. If you need to use these databases, you only need to add the database driver and configuration file connection URL to connect, set the connection target and forwarding target in the database, and the node will automatically read the database to establish receiving and forwarding connections after startup.

[0058] 1.6 Node status setting

[0059] (1) State synchronization mechanism

[0060] The user sends a request to node 1. Node 1 obtains the cluster configuration through the configuration file, asynchronously forwards the request to node 2, node 3, node 4, etc., synchronizes the status, and makes all nodes in the cluster have the same receiving and forwarding status at the same time. When the synchronization is successful, the current status information is saved in the database.

[0061] (2) Node removal mechanism

[0062] When a node goes down or requests are blocked, the node can be removed without affecting the request operations of other nodes.

[0063] (2) State rollback mechanism

[0064] Use the database to store the previous state of the node. When a node service request fails, read the previous state of the database and roll back to restore the previous state.

[0065] (3) State recovery mechanism

[0066] Use the database to store node status, connection and forwarding information. When a node fails, it can be restarted to restore the connection status and other status.

[0067] 1.7 Customizable data filtering protocol

[0068] (1) Customized data filtering algorithm

[0069] Users can customize the data filtering algorithm by uploading the data filtering algorithm binary file. The application loads the binary file into the memory, and then obtains the filtering algorithm through the reflection technology of the programming language, so that the application can filter the data that the user wants to filter through the algorithm.

[0070] 1.8 Custom data connection scheme

[0071] (1) Customize data source connection

[0072] The present invention supports multiple data sources, and configures the data source connection protocol TCP / UDP, data connection mode, data source as client or server, etc. through the configuration file

[0073] (2) Customized TCP data sticky packet and half-packet data processing algorithm

[0074] Configure TCP data sticky packet and half packet data processing algorithm through configuration file

[0075] (3) Customize data protocol parsing rules

[0076] Configure data protocol construction and parsing rules through configuration files to parse data.

[0077] 1.9 Startup Script

[0078] Create an application startup script and place the script in the folder where the server system is started. When the server starts, it will automatically run the application, cache middleware Redis, database and message queue middleware, and other applications in a certain execution order.

[0079] 2.0 Monitoring Script

[0080] Write monitoring scripts to monitor the application process name, cache middleware Redis and database process, which are used to start applications such as database, cache middleware Redis, message queue middleware, etc., and have an execution order. Example

[0081] The present invention is an application developed based on the Java open source technology framework, which supports multiple databases, multiple data sources, multiple data protocols, and multiple stream data filtering algorithms. The construction process is to first build the database and cache middleware Redis, then start the data source, start the node application on multiple servers, upload the configuration file, and finally start the data source to send data to perform data fusion filtering on the data.

[0082] (1) Connector configuration: multiple data source connections are established through multithreading. The connector can be configured with multiple connection protocols, TCP or UDP connection, where UDP supports multicast and unicast modes, and TCP connection supports processing data sticky packets and half-packet problems. The present invention provides two data sticky packet and half-packet processing algorithms, one of which is that each frame in the data has a data header and a data length field, and the current data is judged based on the data header and data length to determine whether it is a complete frame of data. If not, it is directly discarded. The second is to support the frame header, frame tail and verification byte, which is used to detect whether the data is a complete frame of data. If it is a sticky packet, it is unpacked for processing. If it is a half packet, it is added to the cache and processed after the arrival of the next frame of data.

[0083] (2) Protocol configuration: configure the protocol name, protocol ID, and protocol sticky packet and half-packet parsing algorithm, and configure the data type and parsing algorithm represented by each byte of the protocol data;

[0084] (3) Filter configuration: Configure a custom algorithm. The present invention provides two algorithms. One is an algorithm that calculates the speed of the Beidou positioning object based on time-series data, and determines whether the current data or object is abnormal based on the speed, and performs abnormal processing. The calculation formula is as follows:

[0085] distence = R * arccos(sin(latitude1) * sin(latitude 2) + cos(latitude1) * cos(latitude 2) * cos(longitude1 - longitude 2))

[0086] v = distence / (time2 – time1)

[0087] Where distance is the distance, R is the radius, latitude1 is the latitude of the first point, latitude2 is the latitude of the second point, longitude1 is the longitude of the first point, longitude2 is the longitude of the second point, v is the speed, time2 is the time of the second point, and time1 is the time of the first point.

[0088] You can also customize the algorithm. If the abnormal data is Gaussian distributed, you can use the Gaussian filtering algorithm to establish a confidence interval, retain the confidence interval data, and filter out erroneous data.

[0089] (4) Configure the difference algorithm: If it is GPS data or Beidou data, you can add configuration options. Due to the use of multi-threading technology, it can support the difference calculation of multiple objects.

[0090] (5) Method configuration in field configuration, parse protocol data, or use the protocol data parsing method provided by the node application.

[0091] The present invention discloses a data fusion filtering method for multi-threaded and multi-link data, the main advantages of which are embodied in the following aspects:

[0092] 1. We have designed a multi-link data fusion solution. Even if the server goes down or the application process disappears, data can be restored within one second, so that customers do not have to worry about power outages. There is also a startup script, which allows you to run the application by turning on the server without too much operation.

[0093] 2. A cluster building solution is designed. Using multi-threading technology, a request is sent to a node, and the same request is sent to other nodes asynchronously through multi-threading. In this way, the problems of cluster configuration file upload, configuration file synchronization, and sending status synchronization can be solved. However, when a server crashes, the application node on this server will disappear, and other nodes will remove this node without affecting the normal operation of other nodes. However, when a node has a request problem, the operation of rolling back the status of all nodes can also be set to make the status of all nodes consistent, thereby reducing operation and maintenance operations.

[0094] 3. A method that supports custom data filtering algorithms is designed. Users can customize the data they want to obtain according to their data needs, and customize their own filtering methods by uploading binary files and configuration files. It is simple and effective.

[0095] 4. A simple service building method that supports connecting to multiple data sources is designed. Using multi-threading technology, one node can connect to multiple data sources. By configuring the basic information of multiple data sources, forwarding destinations, routing tables and other writing operations, and uploading configuration files, the application nodes required by users can be customized and saved in the database. When the node is restarted, the last configured connection information is still used, and there is no need to upload the configuration file again.

[0096] 5. A universal solution for data protocol parsing is designed. By customizing the configuration file protocol, the function of application nodes to process data protocols is universal, reducing the developers' research and development of new protocols. In addition, only one configuration file needs to be uploaded to build an environment for custom data protocol parsing algorithms, thereby reducing the number of user-defined data protocol operations.

[0097] 6. Designed an overall architecture, including cache middleware Redis, database and node application, and supports clients written in multiple languages ​​to access and obtain data.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-link data receiving fusion method based on multi-threading, characterized in that: The method is based on a fusion system, which consists of multiple nodes to form a cluster. Each node is connected to three targets. The first target is a data source, which is used to receive data. The second target is a database, which is a relational database and is used to record cluster status. The third target is a cache middleware Redis, which is used to execute a data fusion script protocol. The method includes: In each node, the receiver form, data protocol form and data filtering algorithm are defined through a custom configuration file, and multiple threads are established through thread pool technology. Each thread establishes a receiver and runs data receiving and data processing services at the same time. The data processing services of each receiver will not affect each other. After parsing, the data from multiple data sources are aggregated to perform data deduplication and data filtering operations. Each node is an independently running node, and each node uses multi-threading technology to connect to multiple data sources and run in parallel. Each node deduplicates the data received by the data source, and then calculates the data distance and reception time difference between the two nearest points to determine whether it is erroneous data. If so, it is filtered and finally the processed data is sent to the cache middleware Redis. The script technology and SET collection of the cache middleware Redis are used to perform data deduplication and data transmission, and the cached outdated data is destroyed regularly. The cache middleware Redis is set outside the application cluster. Each node in the cluster connects to the cache middleware Redis through the TCP protocol and sends data to the SET set of the cache middleware Redis. If the SET set does not exist, the data is sent to the blocking queue. The cache middleware Redis supports multiple language connections and can consume blocking queue data. Each node is deployed on a different server. Each node is a process in the server. The cluster is built on multiple servers, and multiple servers are distributed in multiple computer rooms. If a server goes down or restarts, the process disappears and the cluster loses a node. However, other nodes are still running and data will not be lost. All nodes receive the same data, so the sent data is also the same. Even if a node server fails, it does not affect other nodes from receiving and sending data. Due to the cache middleware Redis script technology and the non-repetitive nature of the SET set, the data of all nodes is sent to this SET set to determine whether it exists. If it exists, it means that it has been received. If it does not exist, it means that the data has not been received. Therefore, when a node hangs up, the data can be resumed within 1 second, and the data is unique. The data received by each node has data time. First, each node will filter out the data with disordered time. However, the time sent by each node to the cache middleware Redis is different. Therefore, there are two solutions to customize the script protocol of the cache middleware Redis. The first solution is that because the data of each node is ordered, you only need to set the SET collection in the cache middleware Redis, send the data of all nodes to the SET collection, and use the non-repeatability of the SET collection to determine whether the data is received. If there is no data in the SET collection, the data will be cached in the SET collection and sent to the blocking queue. Because the queue has the feature of first-in-first-out, the data that is finally consumed is ordered. The second solution is to The permission Key is set in the middleware Redis, and has a validity period of 5 seconds. The value of the Key is the IP address of the node. The rule for assigning this permission is that the node that sends data first has this permission, that is, the value of this Key is the IP address of the node that sends data first. Then this node will have 5 seconds to send data to the SET collection, and then determine whether the collection receives the same data. If not, the data is cached in the SET collection, and then the data is sent to the blocking queue; after 5 seconds, an empty Key is recreated. If any node sends data first, the IP address of this node will be assigned to this Key. Since the data of each node is time-series, the data finally collected in the queue is time-series; The user sends a request to node 1. Node 1 obtains the cluster configuration through the configuration file, asynchronously forwards the request to other nodes, and synchronizes the status so that all nodes in the cluster have the same receiving and forwarding status at the same time. When the synchronization is successful, the current status information is saved in the database. When a node crashes or the request is unavailable, the node is removed without affecting the request operations of other nodes. If it is not a crash or the request is unavailable, when a node service request fails, in order to make all nodes consistent, the previous state of the node stored in the database is used to read the previous state of the database and restore the previous state of the node. The database is used to store node status, connection and forwarding information. When the node hangs up, it can be restarted to restore the connection status and other states.

2. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: The relational databases are Oracle, MySQL, DAMO or DATABASE.

3. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: Multiple data sources send the same data to each node at the same time. Just configure the IP addresses and ports of all nodes in the application configuration file, use server sockets to write rules and connect with commas to write the node cluster configuration.

4. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: Through the configuration file data protocol format configuration, configure the protocol name, protocol ID, and the byte array of the protocol, and use the programming language reflection technology to configure the protocol data in the configuration file. Each byte or each byte array parsing method name, use the reflection mechanism and mapping mechanism of the programming language to map the name to an executable method for the byte data, and parse the data through these methods.

5. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: The user customizes the data filtering algorithm by uploading the binary file of the data filtering algorithm. The application loads the binary file into the memory and then obtains the filtering algorithm through the reflection technology of the programming language. In this way, the application can filter the data that the user wants to filter through the algorithm.

6. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: The user customizes the data source connection, configures the data source connection protocol TCP / UDP, data connection mode, and data source as client or server through the configuration file, configures the TCP data sticky packet and half-packet data processing algorithm through the configuration file, configures the data protocol construction and parsing rules through the configuration file, and parses the data.

7. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: Create an application startup script and place the script in the folder where the server system is started. When the server starts, it will automatically run the application, cache middleware Redis, database and message queue middleware applications, and have an execution order.

8. The multi-link data receiving fusion method based on multi-threading as claimed in claim 1, characterized in that: Write a monitoring script to monitor the application process name, cache middleware Redis and database process, which is used to start the database, cache middleware Redis, message queue middleware application, and has an execution order.

9. The multi-link data receiving and fusion method based on multi-threading according to any one of claims 1 to 8, characterized in that: Through multi-threading, multiple data source connections are established, and the connector is configured with multiple connection protocols, TCP or UDP connection, among which UDP supports multicast and unicast modes, and TCP connection supports processing data sticky packets and half-packet problems; there are two algorithms for processing data sticky packets and half-packet, one of which is that each frame in the data has a data header and data length field, and the data header and data length are used to determine whether the current data is a complete frame of data. If not, it is directly discarded; the second is to support frame headers, frame tails and verification bytes to detect whether the data is a complete frame of data. If it is sticky, it is unpacked for processing. If it is a half packet, it is added to the cache and processed after the arrival of the next frame of data.

10. The multi-link data receiving and fusion method based on multi-threading as claimed in claim 9, characterized in that: Filter configuration includes: configuring custom algorithms, one is to calculate the speed of Beidou positioning objects based on time-series data, and judge whether the current data or object is abnormal based on the speed, and perform abnormal processing. The calculation formula is as follows: distence = R * arccos(sin(latitude1) * sin(latitude 2) + cos(latitude 1)* cos(latitude 2) * cos(longitude1 - longitude 2)) v = distence / (time2 – time1) Where distance is the distance, R is the radius, latitude1 is the latitude of the first point, latitude2 is the latitude of the second point, longitude1 is the longitude of the first point, longitude2 is the longitude of the second point, v is the speed, time2 is the time of the second point, and time1 is the time of the first point; The second is a custom algorithm. If the abnormal data is Gaussian distributed, use the Gaussian filtering algorithm to establish a confidence interval, retain the confidence interval data, and filter out erroneous data.

Images