Global navigation satellite observation data real-time acquisition and management method and system

Abstract

The application discloses a kind of global navigation satellite observation data real-time acquisition and management method and system, using the management mode of distributed database to the process data and result data of global navigation satellite data acquisition Unified warehousing storage, open standard data query interface, for independent research and development software, data sharing exchange provides fast, standard, simple data service mode;At the same time, process data can also be retained, provide the historical traceability ability of data.In addition, the application also uses distributed message queue framework and distributed stream data parallel computing framework, meet the large-scale high-frequency data acquisition communication of large-scale satellite navigation positioning reference station (CORS) networking, mass data analysis and calculation, high concurrency data query request, and the needs such as flexible management of mass navigation satellite observation data after long-term accumulation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of satellite positioning, in particular to a global navigation satellite observation data real-time acquisition and management method and system. BACKGROUND

[0002] With the rapid development of multi-constellation multi-frequency satellite systems, the operation of China's Beidou-3 global satellite navigation system and the surge in demand for mass applications of public location services, all types of users have put forward more urgent requirements for faster, more accurate and more reliable location services. As the most important component of spatial data infrastructure, satellite navigation and positioning reference stations (CORS) can provide accurate and reliable spatial positioning services for all types of users. In the past decade, the Ministry of Natural Resources, China Earthquake Administration and various provincial surveying and mapping departments or units have established CORS of different levels, different accuracies and different functions, providing high-precision positioning support services in national economic construction. With the rapid rise of new generation information technologies and applications such as mobile Internet and Internet of Things, the coverage range and service field of high-precision positioning applications continue to deepen, and the unified collection, unified management, and standardized, rich and flexible data services of CORS data information are the inevitable trend of the enhancement of satellite navigation and location service capabilities in the era of Internet big data.

[0003] The important content of CORS management and maintenance is the collection, storage and management of satellite observation data. With the increasing demand for satellite data in various industries, the current CORS station construction types, construction standards and application scenarios are not the same, and they are basically equipped with more sensor equipment for observing various types of data (such as GNSS data, meteorological information data, etc.). At present, the data collection and management of CORS in each province mainly relies on the software system provided by the sensor equipment supplier, and often multiple software systems are parallel, which faces the following problems: 1) Different data formats: In addition to the different data formats caused by the different contents of the data itself, each equipment manufacturer also stores the same type of data in a self-defined file format, such as Trimble using T02 format, South Surveying and Mapping using STH format to store observation data, etc.; 2) Data content loss: Although the Radio Technical Commission for Maritime Services (RTCM) has formulated a differential global navigation system service standard to facilitate the exchange and processing of observation data, different manufacturers have made some changes to the RTCM part of the text information in order to adapt to their own algorithm and software compatibility, so even if the parsed and stored data is encoded as standard data, there may still be differences between different manufacturers' data; 3) Complex use process: Data users always need a complex data conversion and extraction process, and different preprocessing methods are used according to the data type and data source to convert the data into a standard or internal acceptable format before the data can be used for application calculation, which is very not conducive to the wide use of satellite data in the future; 4) Not supporting real-time calculation: Since the data collection process is a closed black box process in the equipment manufacturer's software, the software only provides static result data, and does not fully open the collection process data, which may cause some application functions to be difficult to realize or not to be realized, such as real-time precise point positioning (PPP) calculation which must collect and analyze CORS data quickly and correctly to obtain effective observation values and then use the non-difference model to realize positioning; 5) Management confusion: In terms of data management, the existing navigation satellite observation data is stored in the file system in the form of files, and the data is indexed through the file path structure and file name, and the file directory and file name indexing method of multiple CORS data are independent of each other, without database management, and without unified data directory and efficient data query service. It can be seen that the existing data management capability is not conducive to the long-term management of CORS, and to a large extent, it limits the expansion of satellite data services and large-scale application development. SUMMARY

[0004] The present application aims at the problems of data variety, use scene variety, different data storage format standards of equipment manufacturers, closed data collection process, inconvenient data use, complex preprocessing process, low efficiency and the like mentioned in the background art, and the current situation of low data management efficiency and inability to meet the efficient query access requirements of large-scale applications due to the lack of satellite data library management, and proposes a global navigation satellite observation data real-time collection and management method and system, the main features of which are: using the management mode of a database to store the process data and result data of global navigation satellite data collection in a unified manner, and opening a standardized data query interface, thereby changing the traditional commercial software coding and independent service mode, and providing a fast, standard and simple data service mode for self-developed software and data sharing and exchange. At the same time, the process data is retained, the historical traceability of the data is provided, more flexible data applications can be supported, and the data application process is simplified.

[0005] In addition, considering the development trend of the nationwide satellite navigation positioning reference station "one network", in order to meet the requirements of large-scale high-frequency data collection communication, massive satellite data analysis and calculation, high-concurrency navigation satellite data query request and elastic management of massive observation data accumulated for a long time, the present application also improves the global navigation satellite data collection and management method and system by using a distributed big data framework.

[0006] The specific technical solutions of the present application include:

[0007] Scheme one: the application discloses a global navigation satellite observation data real-time collection and management method, mainly comprising the following steps: receiving satellite observation data streams based on RTCM3.2 standard sent by N CORS stations simultaneously and uninterruptedly according to TCP / IP protocol; parsing the observation data streams into multiple independent original observation data, each original observation data records all fields of one message type; establishing a stream task to correct the content of each original observation data by parallel processing, obtaining first observation data; filtering the first observation data by using a data checking method in the stream task, obtaining second observation data; establishing a cache area to cache the second observation data, the cache area is divided into M cache partitions, the second observation data of the same CORS station is stored in the same cache partition, and a data consumption interface is provided for each cache partition; sending a data acquisition request to each data consumption interface, packing the acquired second observation data according to a preset rule to establish multiple parallel computing units; starting a parallel computing task to parse and calculate the second observation data in the parallel computing units, obtaining the message fields recorded in the data area and the identification area of the second observation data; writing the message fields and the binary original text of the data area and the identification area of the second observation data into a distributed database for storage management, the message fields and the binary original text segment constitute process data, and a first query interface of the process data is provided; reading the message fields from the distributed database at regular intervals and generating static observation files, writing the files into the distributed database for storage management, and providing a second query interface of the files; the first query interface and the second query interface are defined based on HTTP RESTful specification, and N and M are natural numbers greater than 1.

[0008] As a preferred scheme, the distributed database organizes and manages data by station, message type and time, specifically as follows: the distributed database has multiple station sets station distinguished by CORS stations; the station set station includes multiple message type sets type, and all process data of the same message type collected at different times are stored in the same message type set type; the station set station also includes an observation file set, and the observation file set is divided into a files set and a chunks set; the files set is used to store metadata describing static observation files; and the chunks set stores specific contents of the static observation files in the form of binary data.

[0009] As a preferred scheme, the data structure of the process data comprises: a message type typeId, a reference station station number stationId, an observation data collection time time, a binary original text of a data area and an identification area of second observation data, and a message field parsed from the data area and the identification area of the second observation data.

[0010] As a preferred solution, the process data in the message type set type is stored in order of the time when it is collected.

[0011] As a preferred solution, the first query interface is defined as:

[0012] http: / / host[:port] / path?{stationId=value&typeId=value&startTime=value&endTime=value&interval=value};

[0013] Wherein, host[:port] is the server address and port of the query service, path is the path of the service, and {} indicates the input parameters of the interface; the input parameters include the reference station station number stationId, the message type number typeId, the data sampling interval interval, the start time startTime and the end time endTime of the query time range.

[0014] As a preferred solution, each record in the files set can be a file or a folder; the data structure of the files set includes: the unique identity of the file or folder _id, the identity of the upper folder of the file or folder _parentId, the file name or folder name name, the observation data start time of the file record dataStartTime, the file data accumulation duration range, the data sampling interval interval, the total size of the file length, and the file chunk size chunkSize; if the record is a folder, the values of length, chunkSize, dataStartTime, range, and interval are all empty; when storing the file, the distributed database divides the static observation file with a file data size greater than chunkSize into multiple data blocks and stores them in the chunks set; the file in the files set corresponds to at least one data block; the data structure of the chunks set includes: the unique identity of the file to which the data block belongs _id, the position identifier of the data block in the file n, and the serialized binary data of the data block data.

[0015] As a preferred solution, the file or folder is virtually established into a folder system by associating its unique identity _id and the upper folder identity _parentId.

[0016] As a preferred solution, the second query interface is defined as:

[0017] http: / / host[:port] / path?{stationId=value&dataStartTime=value&range=value&interval=value};

[0018] Wherein, the host[:port] is the server address and port of the query service, the path is the path of the service, and the {} indicates the input parameters of the interface; the input parameters include the reference station station number stationId, the data sampling interval interval, the data accumulation time length recorded by the file range, and the data start time recorded by the file dataStartTime.

[0019] As a preferred solution, the way of generating the static observation file is that: the first query interface reads the message field corresponding to the required message type in the required station within the required time range from the distributed database, samples the data at the sampling interval, returns the result data, and generates the target static observation file. Preferably, the static observation file is a RINEX file.

[0020] As a preferred solution, the observation data stream is parsed into multiple independent raw observation data, specifically including: for the observation data stream of each station, sequentially reading the binary bytes, and finding the fixed guide symbol according to the observation data transmission format described in the RTCM3.2 standard, taking the fixed guide symbol as the head, calculating the total length of the prefix symbol, the reserved field, the data area length and the check area description, defining the data boundary with the guide symbol and the total length, and splitting the observation data stream into multiple raw observation data.

[0021] As a preferred solution, a custom identification area is added at the end of each raw observation data for content modification, specifically including: positioning to the end of the raw observation data according to the RTCM3.2 standard, and adding an identification area with a custom number of bits after the end of the data, marking the content to contain the station number ID of the original observation data source station and the time and serial number of the original observation data received.

[0022] As a preferred solution, the data checking method is CRC checking method.

[0023] As a preferred solution, the serial number x of the second observation data source station is taken, the station serial number is subjected to a modulo operation through formula (1) to obtain the cache partition serial number f(x), and then the observation data is stored in a specific cache partition Part f(x) ,

[0024] f(x)=x%M (1)

[0025] When the number of stations N is less than or equal to M, one cache partition stores the second observation data from the same CORS station; otherwise, one cache partition stores the second observation data from more than two CORS stations.

[0026] As a preferred solution, the obtained second observation data is packaged according to preset rules to establish a plurality of parallel computing units, specifically including: for each data consumption interface, setting a batch processing time interval ΔT1, encapsulating all second observation data from the current data consumption interface within ΔT1 as a unit to form a parallel computing unit; setting a window time interval ΔT2, ΔT2 is a multiple of ΔT1, within the time interval of ΔT2, K parallel computing units are formed, K = ΔT2 / ΔT1; MxK parallel computing units from M data consumption interfaces are packaged as a batch analysis task, MxK parallel computing units are calculated in parallel, and the analysis and calculation of the second observation data are performed; after a certain time T of the current batch analysis task is executed, slide to the next window time, obtain new second observation data and establish a new batch analysis task containing MxK parallel computing units; T is the execution period, and the time length is the average time of completing the analysis and calculation of a batch of parallel computing units.

[0027] As a preferred solution, the parallel computing task is started to analyze and calculate the second observation data in the parallel computing unit, specifically including: defining an analysis function Parse.getData(key) for each message type, the key is the message type, and the value is the second observation data, generating a kv(<key, value>) pair for each second observation data in the parallel computing unit; when the parallel computing task is executed, the corresponding analysis function is called based on the key value, and all kv pairs in the parallel computing unit are analyzed and calculated, and the result of the analysis and calculation is the content of the data area and the identification area in the second observation data. The message field recorded according to the RTCM3.2 standard.

[0028] Solution two: the application also provides a global navigation satellite observation data real-time acquisition and management system, including a data acquisition center, a data analysis center and a distributed database, the data acquisition center is designed based on a distributed message queue technology, and the data analysis center is designed based on a distributed stream data computing framework; wherein:

[0029] The data collection center is configured to include a stream analysis module, a content correction module, a data filtering module and a data cache area; the stream analysis module is used to receive satellite observation data streams based on RTCM3.2 standard sent by N CORS stations simultaneously and uninterruptedly in TCP / IP protocol, and analyze the observation data streams into multiple independent original observation data, each of which records all fields of one message type; the content correction module is used to establish a stream task to correct the content of each original observation data by parallel processing to obtain first observation data; the data filtering module is used to filter the first observation data by data checking method in the stream task to obtain second observation data; the cache area is divided into M cache partitions, and the second observation data of the same CORS station is stored in the same cache area, and each cache partition has a data consumption interface; N and M are natural numbers greater than 1;

[0030] The data analysis center is configured to include a parallel computing unit establishment module, a data content analysis module and a real-time writing module: the parallel computing unit establishment module is used to send a data acquisition request to each data consumption interface, and package the acquired second observation data according to a preset rule to establish multiple parallel computing units; the data content analysis module is used to start a parallel computing task to analyze and calculate the second observation data in the parallel computing units to obtain the message fields recorded in the data area and the identification area of the second observation data; the real-time writing module is used to write the message fields and the binary original text of the data area and the identification area of the second observation data into a distributed database for storage management, and the message fields and the binary original text constitute process data;

[0031] The distributed database has multiple station sets station distinguished by CORS stations; the station set station includes multiple message type sets type, and all process data of the same message type collected at different times is stored in the same message type set type; the station set station also includes an observation file set, which is divided into a chunks set and a files set, the files set is used to store metadata describing static observation files, and the chunks set stores specific contents of the static observation files in binary data form;

[0032] The distributed database also has a first query interface for process data query and a second query interface for static observation file query, and the first query interface and the second query interface are defined based on HTTP RESTful specification.

[0033] Preferably, the distributed message queue technology is Kafka, RabbitMQ or AMQP.

[0034] Preferably, the distributed stream data computing framework is Spark Steam or Storm, Flink.

[0035] Preferably, the distributed database is MongoDB, Cassandra or CouchDB.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] (1) The prior art mostly provides the final product of data collection including RINEX files or self-format files, while the present application collects and records the raw observation data original messages and parsed message fields of the preprocessed satellite observation data in addition to RINEX files, so that the types of data products are more diversified, and the application requirements of real-time and static observation data can be flexibly adapted.

[0038] (2) The present application manages all data in the form of a database, and the management efficiency is superior to the file management mode of the prior art, can support the massive data accumulated after long-term operation of a large-scale station network, can provide a complete data directory for data managers, and ensures the flexibility of data expansion, thereby providing a basis for large-scale station network management.

[0039] (3) The prior art mostly opens the file system access path such as the FTP protocol to provide static observation data download, while the present application can provide efficient and networked data sharing and exchange through the network query interface of the HTTP RESTful standard, greatly accelerates the batch acquisition ability of data, greatly facilitates the development and utilization of navigation and positioning satellite data, and provides a favorable technical foundation for the internet application service of navigation and positioning satellite data.

[0040] (4) The present application realizes autonomous satellite data collection, records the process data such as the raw observation data original messages and parsed message fields of the preprocessed observation data, can transfer relatively raw observation data in real time through the query interface, can avoid the complex data extraction process when using observation data in different application scenarios, improve the efficiency of data use, and realize the computing application with the demand of real-time satellite observation data; at the same time, the relevant problem data can be traced and traced, and then the missing data due to conversion can be supplemented from the database, and the traditional coding mode of commercial software is also broken.

[0041] (5) The application adopts the current mainstream distributed big data framework, including a distributed message queue framework supporting large-scale data communication, a distributed stream data computing framework supporting real-time computing of massive data, and a distributed data supporting reading and writing of the distributed computing framework and supporting storage of rapidly growing observation data, so that the processing efficiency and stability of the system can be ensured, the benchmark station (data source node) can be increased indefinitely, and the demand for flexible expansion and large-scale application of the benchmark station network can be met. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Schematic diagram of a global navigation satellite observation data real-time acquisition and management system architecture;

[0043] Figure 2 Schematic diagram of a global navigation satellite observation data real-time acquisition and management method step;

[0044] Figure 3 Kafka-based distributed data acquisition method architecture diagram;

[0045] Figure 4 Spark-based distributed data analysis method architecture diagram;

[0046] Figure 5 Part of the content of the RINEX file;

[0047] Figure 6 Distributed database logical storage structure. DETAILED DESCRIPTION

[0048] The application provides a global navigation satellite observation data real-time acquisition and management method and system, which acquires satellite observation data stream from a CORS reference station (referred to as a CORS station or a reference station), analyzes and splits the observation data stream into original observation data, corrects the content of the original observation data and filters the data, analyzes the data content of the observation data after correction and filtering, directly stores the analyzed data fields (i.e. message fields) into a database, reads the data fields from the database at regular intervals, generates RINEX static observation files or other self-defined static observation files according to a certain rule, and stores these static observation files in the database management.

[0049] The technical solutions described in the application will be further explained and described in combination with specific embodiments and the accompanying drawings.

[0050] In combination with Figure 1As shown, embodiment 1 discloses a global navigation satellite observation data real-time acquisition and management system, mainly including a distributed data acquisition center, a distributed data analysis center and a distributed database. The data acquisition center is designed based on distributed message queue technology, such as Kafka, RabbitMQ, AMQP, etc., mainly including several data processing modules such as stream analysis module, content correction module, data filtering module, and data cache area. The data analysis center is designed based on distributed stream data computing framework, such as Spark Steam, Storm, Flink, etc., mainly including parallel computing unit, data content analysis module and real-time writing module. The distributed database can use, for example, MongoDB, Cassandra, CouchDB, etc., mainly used to store filtered observation data and its analysis results (i.e. message field), and also contains static observation files generated by subsequent database operations. Figure 1 The main components of the global navigation satellite observation data real-time acquisition and management system and the data processing content and process of each part are shown, and the specific description is as follows:

[0051] The data acquisition center can support the reception of large-scale CORS reference station uninterrupted observation data stream, and a series of parallel tasks are established for the received satellite observation data stream to preprocess the data. The reference station continuously sends satellite observation data stream based on RTCM3.2 standard to the data acquisition center according to TCP / IP protocol, that is, a series of observation data connected end to end. The contents of preprocessing mainly include: first, the satellite observation data stream is split into independent raw observation data in the stream analysis module; since the received satellite observation data sometimes has the case of missing CORS reference station station number value in the data content, it is necessary to correct the content of each raw observation data in the content correction module to ensure that the subsequent processed observation data contains valid station number and collection time, which are necessary key information when doing data management; then, the corrected observation data (which can be called "first observation data") is filtered in the data filtering module, that is, through data correctness verification to ensure that the data content is not wrong due to code loss or code error in network transmission. The data cache area has multiple cache partitions, and after the valid observation data (which can be called "second observation data") is verified, the valid observation data is allocated to the multiple cache partitions of the data acquisition center for queuing, waiting for the data analysis center to obtain and process from the cache area of the data acquisition center when there are computing resources.

[0052] The data parsing center is used to parse the preprocessed observation data from the data acquisition center, i.e., the second observation data. The data parsing center requests the preprocessed observation data from the data acquisition center's cache, packages the second observation data according to certain rules to create multiple parallel computing units, and initiates parallel computing tasks in the data content parsing module to parse and compute the content of the observation data in all parallel computing units, that is, to parse the content of the data area and identifier area in the second observation data. The parsing result is many message fields described by the observation data according to the RTCM 3.2 standard. The parsing result and the parsed second observation data are collectively referred to as "process data." The process data is written to the database for storage in real time, and a query interface for the process data is also provided.

[0053] During normal system operation, message fields can be queried periodically from the distributed database to generate static observation files, which are then written back to the database for storage and management. A query interface for the static observation file data is also provided. All of these query interfaces are defined based on the HTTP RESTful specification.

[0054] Combination Figure 2 to Figure 6 As shown, Example 2 discloses a method for real-time acquisition and management of global navigation satellite observation data, which can be based on the real-time acquisition and management system for global navigation satellite observation data described in Example 1, and mainly includes the following steps:

[0055] Data reception: Define the CORS reference station as the production node for observation data, and its set is S{S1, S2, ..., S}. n The number of stations is N. All stations simultaneously and continuously send satellite observation data streams to the data acquisition center, which receives these data streams. A satellite observation data stream is typically a continuous, unbounded string of binary bytes, containing multiple consecutive, unbounded observation data points.

[0056] Stream parsing: The data acquisition center first needs to perform stream parsing, which involves splitting the observation data stream into multiple independent raw observation data streams. For each station's observation data stream, binary bytes are read sequentially, and a fixed guide sign is located according to the observation data transmission format described in the RTCM3.2 standard. Using the fixed guide sign as the header, the total length of the prefix sign, reserved fields, data area length, and check area description bytes is calculated. The data boundaries are defined using the guide sign and the total length, thus splitting the observation data stream into multiple raw observation data streams.

[0057] Content correction: After splitting, the data acquisition center establishes a stream task to process the observation data in parallel according to the number of observation data that need to be processed at the same time (usually the number is greater than N), and performs content correction on each observation data. According to the RTCM3.2 standard, the observation data is positioned to the end of the data (i.e. the end of the check area byte), and a self-defined 32-bit identification area is added after the end of the data to obtain the first observation data, as shown in Table 2. The marking content includes the station number ID of the data source station, the time when the data is received, and the serial number, and the time record is accurate to seconds.

[0058] Table 1 RTCM3.2 standard data transmission frame structure

[0059] Name Number of bits Unit Range Description Prefix symbol 8 — — Fixed pilot symbol 11010011 Reserved field 6 — — Reserved field, set to 000000 Data area length 10 byte 0～1023 Data area — — Total length determined by data area length Check area 24 1 — CRC24Q check

[0060] Table 2 modified observation data transmission frame structure

[0061]

[0062] Data filtering: After content correction, the observation data in the stream task is filtered using the CRC check method to judge the correctness of the data. CRC check method is a channel coding technology for generating a short fixed bit check code according to network data packet, mainly used to check possible errors after data transmission. From the first bit of the prefix symbol of the observation data to the last bit of the data area, the CRC24Q check formula is used to generate the check bit sequence. The calculated check sequence is compared with the check code in the check area of the current observation data. If they are consistent, the check is passed, and the observation data is considered complete and can be correctly parsed. If they are not consistent, the check is not passed, and the observation data content is considered missing and cannot be correctly parsed, so the data is discarded.

[0063] Data buffering: The observation data obtained after filtering is stored in the buffer area, waiting for the data analysis center to request data from the buffer area when there is idle computing resource. The buffer area is specifically divided into M buffer partitions, and its set is Part{Part1, Part2, … Part M}, which is used to receive filtered observation data from N reference stations. When buffering, the second observation data should be evenly distributed to each buffer partition to maintain the balance of computing resources. Since the reference stations send data at the same frequency, under ideal conditions, the number of buffer partitions can be equal to the number of reference stations (i.e. M=N), and the data from a station is sent to the same buffer partition. However, when the number of stations is large, the actual running efficiency of the distributed message queue may be reduced, and it is not always possible to one-to-one correspondence between the buffer partition and the station, so the following method can also be used to cache the number of partitions, that is:

[0064] The sequence number x of the observation data source site is taken, the site sequence number is subjected to a modulo operation through formula (1), a cache partition sequence number f(x) is obtained, and then the observation data is stored in a specific cache partition Part f(x) .

[0065] f(x) = x % M (1)

[0066] Wherein, M is equal to the optimal number of partitions in the actual operation of the message queue (usually an experimental experience value in different computing resource environments). When the number of sites N <= M, one cache partition stores observation data from one reference site, otherwise one cache partition may contain observation data from multiple sites.

[0067] The above steps are completed in the data collection center based on the distributed message queue technology. The data collection center here uses the Kafka stream data processing platform as shown in Figure 3 , but the actual application is not limited to this kind of stream data processing platform, and RabbitMQ, AMQP processing platform can also be used. The main work of the data analysis center will be further introduced as shown in Figure 4 , which takes the Spark Steam designed based on the distributed stream data computing framework as an example. Similarly, the actual application is not limited to this kind of data architecture, for example, it can also be Storm, Flink, etc.

[0068] Establish a parallel computing unit: the data analysis center actively obtains observation data (i.e. consumption) from the data collection center. Each cache partition opens a data consumption interface (referred to as "consumption interface"), and the data analysis center simultaneously sends data acquisition requests to M consumption interfaces. After acquiring the data, the data can be packaged according to the following rules to generate a basic computing unit RDD (i.e. programmable data unit) of a group of parallel computing tasks. For each consumption interface, set a batch processing time interval ΔT1, encapsulate all observation data from the current consumption interface within this time unit to form an RDD. Set a window time interval ΔT2, ΔT2 must be a multiple of ΔT1, within the time interval of ΔT2, form K RDDs, K = ΔT2 / ΔT1. Spark packages MxK RDDs from M consumption interfaces into a batch analysis task, and performs parallel computing on MxK RDDs to execute the analysis of the observation data area content. After a certain time T (also known as execution period) of the current batch analysis task is executed, slide to the next window time, acquire new observation data and establish a new batch analysis task containing MxK RDDs. The time length of T is set to be equal to the average time of completing the analysis of a batch of RDDs.

[0069] Data content analysis: According to the current batch of MxK RDD, the observation data contained in the RDD is analyzed and calculated in parallel. According to the RTCM3.2 standard, the data area of the observation data record records the field value of different message types (as shown in Table 3), and each observation data only records all fields of one message type. Take the message type recorded in the observation data as the key, generate a kv(<key, value>) pair for each observation data in the RDD, and the value is the data area and identification area of the observation data. Since each message type contains different fields (as shown in Table 4), the analysis methods of different message types are different, so a parsing function Parse.getData(key) is defined for each message type.

[0070] Table 3 Message type table in RTCM3.2 standard (only part of the message)

[0071] Message type Message name Number of bytes a ]] Description 1～100 Test message - - 1001 GPS RTK L1 observation 8.00 + 7.25 x Ns 1002 Extended GPS RTK L1 observation 8.00 + 9.25 x Ns 1003 GPS RTK L1 & L2 observation 8.00 + 12.625 x Ns 1004 Extended GPS RTK L1 & L2 observation 8.00 + 15.625 x Ns 1005 RTK reference station ARP 19 1006 RTK reference station ARP with antenna height 21 1007 Antenna description 5-36 1008 Antenna description and serial number 6-68 1009 GLONASS RTK L1 observation 7.625 + 8 x Ns 1010 Extended GLONASS RTK L1 observation 7.625 + 9.875 x Ns 1011 GLONASS RTK L1 & L2 observation 7.625 + 13.375 x Ns 1012 Extended GLONASS RTK L1 & L2 observation 7.625 + 16.25 x Ns 1013 System parameters 8.75 + 3.625 x Nm Nm = number of message types broadcast 1014 Network aid station data 14.625 1015 GPS ionosphere correction single difference 9.5 + 3.5 x Ns 1016 GPS geometry correction single difference 9.5 + 4.5 x Ns 1017 GPS geometry and ionosphere combined correction single difference 9.5 + 6.625 x Ns

[0072] Table 4 Field content description of message type 1013 (system parameter message) in RTCM3.2 standard

[0073] Data field name Data field number Data type Number of bits Description Message type number DF002 uint12 12 1013 Reference station ID DF003 uint12 12 MJD day number DF051 uint16 16 UTC day seconds DF052 uint17 17 Number of subsequent messages (Nm) DF053 uint5 5 Number of GPS-UTC leap seconds DF054 uint8 8 Message ID #1 DF055 uint12 12 Message #1 synchronization flag DF056 bit(1) 1 Message #1 transmission interval DF057 uint16 16 Message ID #2 DF055 uint12 12 Message #2 synchronization flag DF056 bit(1) 1 Message #2 transmission interval DF057 uint16 16 ...... ...... ...... ...... Message ID # Nm DF055 uint12 12 Message # Nm synchronization flag DF056 bit(1) 1 Message # Nm transmission interval DF057 uint16 16 Total - - 70 + 29 x Nm

[0074] When performing parallel computing tasks, Spark calls the corresponding parsing function based on the key value (i.e. message type) to analyze and calculate all kv pairs in the RDD. The result of the analysis and calculation is the content of the data area and the identification area recorded in the second observation data according to the RTCM3.2 standard (as shown in Table 5). The message field recorded in the data area and the identification area content, wherein the data area records the message field according to the RTCM3.2 standard.

[0075] Table 5 Analysis result of observation data of message type 1013

[0076]

[0077] After the data analysis center finishes processing, the process data is written to the distributed database in real time, and a standard process data query interface is opened. At the same time, the system also reads the message field from the database at regular intervals, generates observation files, and writes the generated observation files back to the database for storage management, and opens a standard observation file data query interface.

[0078] Next, we will continue to introduce the specific work and methods of the distributed database storage for different categories of data:

[0079] Process data (including the text field and the second observation data) storage: the result of the data content analysis calculation is the text field recorded in the second observation data, which includes the text type, the reference station ID, the observation data collection time recorded in the added identification area, and other attributes. The second observation data in the process data is specifically the binary original text of the data area and the identification area, which is stored in the database in binary original text. To improve query efficiency, a tree structure (i.e. nested collection) of a distributed database MongoDB can be used for modeling, and the database contains multiple station collections (station) differentiated by CORS sites, each station collection further divides the data records into multiple text type collections (type), and all process data is stored in the text type collection as a record object sorted by the time when it is collected. The specific data structure of the storage is as follows:

[0080]

[0081]

[0082] Wherein, typeId is the text type, stationId is the reference station station number, time is the observation data collection time, binary is the binary original text of the second observation data, and attribute1, attribute2 and other attributes are the remaining field results parsed from the observation data.

[0083] As shown in Figure 5 , taking the observation data containing the text type 1013 as an example, the storage of the parsed results in the database is as follows:

[0084]

[0085]

[0086] After the data is warehoused, the text field data and the corresponding binary original text query interface based on the HTTP RESTful specification can be defined, so that users can quickly obtain observation data by sending network HTTP requests. The query interface definition is as follows: http: / / host[:port] / path?{stationId=value&typeId=value&startTime=value&endTime=value&interval=value}

[0087] Wherein, host[:port] is the server address and port where the query service is located, path is the path of the service, and {} indicates the input parameters of the interface, as shown below:

[0088]

[0089] The interface example is as follows:

[0090] http: / / www.shxbdcors.com / apps / dataserver / getFieldData?stationId=0&typeId=1013,1006,1033&date_startTime=20201217T032400&date_endTime=20201218T032359&interval=15.

[0091] RINEX file storage: RINEX file is a commonly used file form for storing satellite observation data. Many post-processing calculations of satellite data use RINEX files as data sources. In order to meet the existing usage habits, the observation data message fields directly stored in the database need to be converted into RINEX files and provided to data users. A RINEX file is usually generated by accumulating multiple message type fields from the same reference station for a certain length of time, and has a certain sampling interval, for example, taking the observation data collected every 30 seconds to generate a RINEX file, which can be seen in Figure 5 The part of the file content of the RINEX file generated by using the observation data of the message types 1006, 1008, 1013, 1033 and 1230 is shown. The RINEX file can be a message field accumulated every 30 seconds for 24 hours, or a message field accumulated every 1 second for 8 hours, which can be generated as needed. According to the required data accumulation length and sampling interval, set a timing task to query the field values from the database through the message field query interface. For example, to generate a 30-second frequency RINEX file with a 24-hour accumulation length, the file needs observation data of five message types 1006, 1008, 1013, 1033 and 1230, then after every 24 hours of observation data is collected, the data fields of the corresponding message types at every 30-second interval within 24 hours are queried. Correspondingly, the database message field data query interface used for querying is as follows:

[0092] https: / / www.shxbdcors.com / apps / getFieldData?stationId=0&typeId=1006,1008,1013,1033,1230&startTime=20201217T032400&endTime=20201218T032359&interval=30

[0093] The RINEX file is generated by calculating the field data returned by the query interface and then stored in the distributed database for persistent storage, which facilitates RINEX file management and quick query.

[0094] The distributed database MongoDB uses two collections to store RINEX files, one for storing file data itself and the other for storing metadata information describing the file. Specifically, one collection is files, which is used to store the metadata of RINEX files, such as file name, source site of the file, total duration of recorded data of the file, etc.; one collection is chunks, which is used to write the file content in binary form to storage. First, extend the site collection described in the "process data storage" section, and add a files collection and a chunks collection to each site collection for storing RINEX files containing observation data of this site. At the same time, the files collection of MongoDB needs to be extended, so that each record in the collection can represent a RINEX file or a folder. Both the file and the folder have their unique ID and the ID of the upper folder. By associating the unique ID and the ID of the upper folder, a virtual folder system is established. By using the virtual folder system, RINEX files of the same type (such as files with the same cumulative duration and sampling interval) are placed in the same folder, which can improve the query efficiency.

[0095] The data structure of the files collection is shown below, which mainly includes the unique ID of the file or folder (_id), the ID of the upper folder of the file or folder (_parentId), the file or folder name (name), the start time of the observation data recorded in the file (dataStartTime), the cumulative duration of the file data (range), the data sampling interval (interval), the total size of the file (length), and the file chunk size (chunkSize). It should be noted that MongoDB database stores file content in chunks. If the data size of the file in the database is greater than the value of chunkSize, the file will be divided into multiple data blocks and stored in the chunk collection.

[0096]

[0097] If the current record is a folder rather than a file, the values of length, chunkSize, dataStartTime, range, and interval are all empty.

[0098] The data structure of the chunk set is shown below, which mainly includes the unique ID of the RINEX file to which the data chunk belongs (_id), the position identification (n) of the data chunk in the RINEX file, and the serialized binary data of the data chunk (data).

[0099]

[0100] After the RINEX file is stored in the database, a RINEX file query interface based on the HTTP RESTful specification can be defined to allow users to quickly query the RINEX file by sending a network HTTP request. The query interface is defined as follows: http: / / host[:port] / path?{stationId=value&dataStartTime=value&range=value&interval=value}

[0101] Wherein, host[:port] is the server address and port where the query service is located, path is the path of the service, and {} indicates the input parameters of the interface, which are as follows:

[0102]

[0103]

[0104] An interface example is as follows:

[0105] https: / / www.shxbdcors.com / apps / dataserver / getRINEX?stationId=0&range=24&interval=30&dataStartTime=20201217T032400.

[0106] Finally, it should be noted that although the embodiments of the present application are described above in combination with the drawings, the present application is not limited to the above specific embodiments and application fields. The above specific embodiments are only illustrative and guiding, but not limiting. Those skilled in the art can make many forms without departing from the scope of the claims of the present application under the guidance of the present specification, which are all within the protection scope of the present application.

Claims

1. A method for real-time acquisition and management of global navigation satellite observation data, characterized in that, The method comprises the following steps: Receiving satellite observation data streams based on RTCM3.2 standard from N CORS stations sent simultaneously and uninterruptedly through TCP / IP protocol; Parsing the observation data streams into multiple independent original observation data, each of which records all fields of one message type; Establishing a stream task to correct the content of each original observation data through parallel processing to obtain first observation data; Filtering the first observation data in the stream task using a data checking method to obtain second observation data; Establishing a cache area to cache the second observation data, the cache area being divided into M cache partitions, the second observation data of the same CORS station being stored in the same cache partition, and a data consumption interface being provided for each cache partition; Sending a data acquisition request to each data consumption interface, and packing the acquired second observation data according to a preset rule to establish multiple parallel computing units; Starting a parallel computing task to parse and calculate the second observation data in the parallel computing units to obtain the message fields recorded in the data area and the identification area of the second observation data; Writing the message fields and the binary original texts in the data area and the identification area of the second observation data into a distributed database for storage management, the message fields and the binary original text segments constituting process data, and providing a first query interface of the process data; Reading the message fields from the distributed database at a regular time and generating static observation files, and writing the files into the distributed database for storage management and providing a second query interface of the files; The first query interface and the second query interface are defined based on HTTP RESTful specification, and N and M are natural numbers greater than 1.

2. The real-time collection and management method of claim 1, wherein, The distributed database organizes and manages data by site, message type and time, specifically as follows: The distributed database has multiple site collections station distinguished by CORS stations; The site collection station includes multiple message type collections type, and all process data of the same message type collected at different times are stored in the same message type collection type; The site collection station also includes an observation file collection, which is divided into a files collection and a chunks collection; the files collection is used to store metadata describing static observation files; and the chunks collection stores the specific contents of the static observation files in binary data form.

3. The real-time acquisition and management method of claim 2, wherein, The process data in the message type collection type is stored in chronological order.

4. The real-time acquisition and management method of claim 2, wherein, The data structure of the process data includes: message type typeId, reference station stationId, observation data collection time time, binary original text of the data area and the identification area of the second observation data, and message fields parsed from the data area and the identification area of the second observation data.

5. The real-time acquisition and management method of claim 4, wherein, The first query interface is defined as: http: / / host[:port] / path?{stationId=value&typeId=value&startTime=value&endTime=value&interval=value}; Wherein, host[:port] is the server address and port where the query service is located, path is the path of the service, and {} indicates the input parameters of the interface; the input parameters include the reference station station number stationId, the message type number typeId, the data sampling interval interval, the start time startTime and the end time endTime of the query time range.

6. The real-time collection and management method of claim 2, wherein each record of the files set is a file or a folder; the data structure of the files set comprises: the unique identity of the file or folder _id, the identity of the upper folder of the file or folder _parentId, the file name or folder name name, the observation data start time of the file record dataStartTime, the file data accumulation time range, the data sampling interval interval, the total file size length, and the file chunk size chunkSize; when storing the file, the distributed database divides the static observation file with a file data size greater than chunkSize into multiple data blocks and stores them in the chunks set; the file in the files set corresponds to at least one data block; if the record is a folder, the values of length, chunkSize, dataStartTime, range, and interval are all empty. The data structure of the chunks set comprises: the unique identity of the file to which the data block belongs _id, the position identifier of the data block in the file n, and the serialized binary data of the data block data. The file or folder is virtually connected to the upper folder through the association of its unique identity _id and the upper folder identity _parentId to establish a virtual folder system. The second query interface is defined as:

7. The real-time acquisition and management method of claim 6, wherein, http: / / host[:port] / path?{stationId=value&dataStartTime=value&range=value&interval=value}; 8. The real-time collection and management method of claim 6, wherein, Wherein, host[:port] is the server address and port where the query service is located, path is the path of the service, and {} indicates the input parameters of the interface; the input parameters include the reference station station number stationId, the data sampling interval interval, the data accumulation time range of the file range, and the data start time of the file dataStartTime. ​ ​ ​ 9. The real-time collection and management method of claim 1, wherein, The method for generating the static observation file comprises the following steps: reading, through a first query interface, the message fields corresponding to the required message types in the required stations within the required time range from a distributed database, sampling the data at a sampling interval, returning the result data, and generating a target static observation file.

10. The real-time collection and management method of claim 1, wherein, The static observation file is an RINEX file.

11. The real-time collection and management method of claim 1, wherein, The method for parsing the observation data stream into multiple independent raw observation data comprises the following steps: for the observation data stream of each station, sequentially reading binary bytes, and finding a fixed guide symbol according to the observation data transmission format described in the RTCM3.2 standard, taking the fixed guide symbol as a header, calculating the total length of the prefix symbol, the reserved field, the data area length, and the check area description, defining the data boundary with the guide symbol and the total length, and splitting the observation data stream into multiple raw observation data.

12. The real-time collection and management method of claim 1, wherein, The method for adding a custom identification area at the end of each raw observation data for content modification comprises the following steps: According to the RTCM3.2 standard, the end of the raw observation data is located, and a custom identification area with a certain number of bits is added after the data, which marks the content including the station number ID of the original observation data source station and the time and serial number of the original observation data received.

13. The real-time collection and management method of claim 1, wherein, The data check method is a CRC check method.

14. The real-time collection and management method of claim 1, wherein, Taking the sequence number x of the second observation data source station, performing a modulo operation on the station sequence number through formula (1) to obtain the cache partition sequence number f(x), and then storing the observation data to a specific cache partition Part f(x) , f(x)=x%M (1) When the number of stations N is less than or equal to M, a cache partition stores the second observation data from the same CORS station; otherwise, a cache partition stores the second observation data from two or more CORS stations.

15. The real-time collection and management method of claim 1, wherein, The method for packing the obtained second observation data according to a preset rule to establish multiple parallel computing units comprises the following steps: For each data consumption interface, a batch processing time interval ΔT1 is set, all second observation data from the current data consumption interface within the ΔT1 time interval is encapsulated to form a parallel computing unit, a window time interval ΔT2 is set, ΔT2 is a multiple of ΔT1, and K parallel computing units are formed within the ΔT2 time interval, K=ΔT2 / ΔT1; M×K parallel computing units from M data consumption interfaces are packed into a batch analysis task, and the M×K parallel computing units are calculated in parallel to perform the analysis and calculation of the second observation data. After a certain time T of executing the current batch analysis task, the next window time is slid to obtain new second observation data and establish a new batch analysis task containing M×K parallel computing units; T is the execution period, and the length of time is the average time of completing the analysis and calculation of a batch of parallel computing units.

16. The real-time collection and management method of claim 1, wherein, The method for starting a parallel computing task to analyze and calculate the second observation data in the parallel computing unit comprises the following steps: A parsing function Parse.getData(key) is defined for each message type, the key is the message type, and the value is the second observation data, each second observation data in the parallel computing unit generates a kv(<key, value>) pair; when executing the parallel computing task, the corresponding parsing function is called based on the key value, and all kv pairs in the parallel computing unit are parsed and calculated, and the result of the parsing and calculation is the message field recorded according to the RTCM3.2 standard in the data area and the identification area of the second observation data.

17. A real-time acquisition and management system for global navigation satellite observation data, characterized by, The data acquisition center is designed based on distributed message queue technology, and the data analysis center is designed based on a distributed stream data computing framework; wherein: The data acquisition center is configured to include a stream parsing module, a content correction module, a data filtering module and a data cache area; the stream parsing module is used to receive satellite observation data streams based on the RTCM3.2 standard sent simultaneously and uninterruptedly from N CORS stations based on TCP / IP protocol, and parse the observation data streams into multiple independent original observation data, each of which records all fields of one message type; the content correction module is used to establish a stream task to correct the content of each original observation data by parallel processing to obtain first observation data; the data filtering module is used to filter the first observation data in the stream task by using a data checking method to obtain second observation data; the cache area is divided into M cache partitions, and the second observation data of the same CORS station is stored in the same cache area, and each cache partition has a data consumption interface; N and M are natural numbers greater than 1; The data analysis center is configured to include a parallel computing unit establishment module, a data content parsing module and a real-time writing module: the parallel computing unit establishment module is used to send a data acquisition request to each data consumption interface, and to package and process the acquired second observation data according to a preset rule to establish multiple parallel computing units; the data content parsing module is used to start a parallel computing task to parse and calculate the second observation data in the parallel computing unit to obtain the message field recorded in the data area and the identification area of the second observation data; the real-time writing module is used to write the message field and the binary original text of the data area and the identification area of the second observation data into the distributed database for storage management, and the message field and the binary original text segment constitute process data; The distributed database has a plurality of station sets station distinguished by CORS stations; the station set station includes a plurality of message type sets type, and all process data of the same message type is stored in the same message type set type collected at different times; the station set station further includes an observation file set, the observation file set is divided into a chunk set and a file set, the file set is used for storing metadata describing static observation files, and the chunk set stores specific contents of the static observation files in the form of binary data; The distributed database further has a first query interface for process data query and a second query interface for static observation file query, and the first query interface and the second query interface are defined based on an HTTP RESTful specification.

18. The global navigation satellite system observation data real-time acquisition and management system according to claim 17, wherein, The distributed message queue technology is Kafka, RabbitMQ or AMQP; the distributed stream data computing framework is SparkSteam or Storm, Flink; and the distributed database is MongoDB, Cassandra or CouchDB.

Images