A method and system for atmospheric science data transmission based on HDFS

Abstract

The present invention discloses an atmospheric science data transmission method and system based on HDFS, which relates to the field of computer application technology. The method comprises: a user client sends a data access request to a data access protocol converter; the data access protocol converter converts the data access request into a query instruction, and sends the query instruction to a back-end program client; the back-end program client sends the query instruction to a back-end program server; the back-end program server obtains the query result from the distributed file system according to the query instruction and sends the query result to the back-end program client; the back-end program client performs protocol conversion processing on the query result to obtain a processing result; and the processing result is sent to the user client. The present invention can make data storage and management more convenient and reduce the difficulty of data sharing.

Description

[0001] This application is a divisional application. The application number of the original application is 202111233500.6, the application date is October 22, 2021, and the name of the invention is "A data transmission method and system". Technical Field

[0002] The present application relates to the field of computer application technology, and more specifically, to an HDFS-based atmospheric science data transmission method and system. Background Art

[0003] Atmospheric science data is often stored in the Network Common Data Form (NetCDF) or Hierarchical Data Format (HDF). These files, as self-describing binary data formats, are often used for grid data storage in atmospheric science. With the improvement in the accuracy of various meteorological or oceanographic numerical models and the increase in observation methods and observation points, the volume of atmospheric science data is showing a growing trend. Therefore, data storage needs to provide an adequate and highly scalable storage environment for atmospheric environmental data, while also providing a reliable redundant backup strategy for the data. On the other hand, because NetCDF and HDF files provide a very flexible framework, their formats can be customized by users according to their needs. The lack of uniformity between data increases the difficulty of transmission and sharing.

[0004] Hadoop is a highly successful distributed systems framework in the industrial sector. It is an open-source Java implementation of Google's Google File System (GFS). Hadoop provides reliable solutions for data redundancy, distributed cluster node management, and load balancing. The framework consists of two key components, one of which is the Hadoop Distributed File System (HDFS).

[0005] The Open-source Project for a Network Data Access Protocol (OPeNDAP) is a widely used data transmission protocol in atmospheric science. It is an extension of the HTTP protocol. This protocol enables online access to structured numerical data stored in cloud-based or local HDF, NetCDF, and other data files. OPeNDAP establishes a Common Data Model (CDM) for structured numerical data, enabling efficient and flexible access to data at specific spatial and temporal points within a dataset.

[0006] The OPeNDAP research organization developed a protocol parser called Hyrax. However, Hyrax does not support integration with HDFS. While the back-end server (BES) provides an interface for accessing remote file systems, its front-end server (OLFS) lacks a clear interface for accessing remote file systems. Furthermore, Hyrax requires indexing data directories. For remote file systems, an XML-formatted index file must be added to each directory of the original data to indicate the current directory's contents. This allows OLFS to correctly return the data source directory page. This means that when files are added or deleted from the file system, these index files must be dynamically updated. However, HDFS, to mitigate issues caused by concurrent writes, does not support file modifications, so updates can only be performed by deleting old indexes and adding new ones. Frequent file operations lead to frequent index additions and deletions, which is cumbersome and consumes system resources, reducing efficiency. Summary of the Invention

[0007] In order to solve the above problems, this application proposes an atmospheric science data transmission method and system based on HDFS.

[0008] In a first aspect, the present application provides an atmospheric science data transmission method based on HDFS, the transmission method comprising:

[0009] The user client sends a data access request to the Data Access Protocol (DAP) converter;

[0010] The DAP converter converts the data access request into a query instruction, and sends the query instruction to a back-end server (BES) client;

[0011] The BES client sends the query instruction to the BES server; the BES server obtains the query result in HDFS according to the query instruction and sends the query result to the BES client;

[0012] The BES client performs protocol conversion processing on the query result to obtain a processing result;

[0013] The processing result is sent to the user client.

[0014] Preferably, the DAP converter converting the data access request into a query instruction comprises:

[0015] The data access request is parsed to obtain several parameters in the data access request; the query instruction is generated according to the several parameters and the prescribed grammatical rules; the query instruction includes a query instruction fragment corresponding to each parameter in the several parameters.

[0016] Preferably, the BES client sends the query instruction to the BES server; the BES server obtains the query result according to the query instruction, and sends the query result to the BES client, including:

[0017] The BES client establishes a point-to-point transmission protocol with the BES server;

[0018] The BES client sends the query instruction to the BES server via the point-to-point transmission protocol;

[0019] The BES server obtains the query result according to the query instruction and sends the query result to the BES client;

[0020] The BES client receives and processes the query result.

[0021] Preferably, the BES client and the BES server establish a point-to-point transmission protocol including:

[0022] The BES client sends a transmission control protocol connection request to the BES server;

[0023] The BES server accepts the request and establishes a transmission control protocol connection;

[0024] The BES client sends a connection request to the BES server to apply for establishing a point-to-point transmission protocol; wherein the connection request includes an identifier representing the requested point-to-point protocol connection;

[0025] The BES server receives the connection request and returns an identifier to the BES client to confirm the request result.

[0026] In a second aspect, the present application provides an atmospheric science data transmission system based on HDFS, the transmission system comprising:

[0027] User client, used to send data access request to DAP converter;

[0028] A DAP converter, configured to convert the data access request into a query instruction, and send the query instruction to the BES client;

[0029] BES client, used to send the query instruction to the BES server;

[0030] A BES server is configured to obtain query results according to the query instruction and send the query results to the BES client;

[0031] The BES client is further configured to perform protocol conversion processing on the query result to obtain a processing result;

[0032] The DAP converter is further configured to send the processing result to a user client.

[0033] Preferably, converting the data access request into a query instruction includes:

[0034] The data access request is parsed to obtain several parameters in the data access request; the query instruction is generated according to the several parameters and the prescribed grammatical rules; the query instruction includes a query instruction fragment corresponding to each parameter in the several parameters.

[0035] Preferably, the BES client sends the query instruction to the BES server; the BES server obtains the query result according to the query instruction, and sends the query result to the BES client, including:

[0036] The BES client establishes a point-to-point transmission protocol with the BES server;

[0037] The BES client sends the query instruction to the BES server via the point-to-point transmission protocol;

[0038] The BES server obtains the query result according to the query instruction and sends the query result to the BES client;

[0039] The BES client receives and processes the query result.

[0040] Preferably, the BES client and the BES server establish a point-to-point transmission protocol including:

[0041] The BES client sends a transmission control protocol connection request to the BES server;

[0042] The BES server accepts the request and establishes a transmission control protocol connection;

[0043] The BES client sends a connection request to the server to apply for establishing a point-to-point transmission protocol; wherein the connection request includes an identifier representing the requested point-to-point protocol connection;

[0044] The BES server receives the connection request and returns an identifier to the BES client to confirm the request result.

[0045] In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the pipeline digital management method is executed.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer-readable instructions, which, when a computer reads and executes the computer-readable instructions, enables the computer to execute a pipeline digital management method.

[0047] The present invention proposes a data transmission method and system, which solves the problem that Hyrax does not support docking with HDFS, makes data storage and management more convenient, and reduces the difficulty of data sharing.

[0048] Description of the attached drawings

[0049] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a schematic diagram of the application of the technical solution provided in the embodiments of this application;

[0051] Figure 2 A schematic diagram of the method provided in the embodiments of the present application;

[0052] Figure 3 A schematic diagram of a data access protocol converter provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the system provided in the embodiments of the present application;

[0054] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0056] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0057] Figure 1 This is a schematic diagram of the application of the technical solution provided in the embodiment of this application. Figure 1 As shown, the user can input the data to be queried. The query method of this solution can combine the OPeNDAP protocol and HDFS through the two interfaces of dap and gateway, and can query the data stored on HDFS. In addition, it should be noted that in this application, atmospheric environment data is mainly used as an example for description. It is understandable that atmospheric environment data can be equivalently replaced with data in various other possible business scenarios.

[0058] Figure 2 Schematic diagram of the method provided in the embodiment of this application. Figure 2 As shown, in one possible embodiment, the data transmission method of the present application includes:

[0059] S201: The user client sends a data access request to the DAP converter.

[0060] In a possible implementation, the data access request sent by the user client to the DAP converter is a URL request. The specific format is:

[0061] http: / / <hostname> : <port> / <space> / <path> . <type> ？ <constraints>

[0062] Among them, hostname identifies the requested IP address, port identifies the requested port, space identifies the data access method, which can include local access and remote access, path identifies the access path of atmospheric science data on HDFS, and constraints identifies the parameters required to perform the operation.

[0063] For example, a request format sent by a user client can be specifically exemplified as follows:

[0064] http: / / 192.168.1.1:10000 / gateway / nc / data / hello.nc.nc?pm25[0:1:443][0:1:381]

[0065] Among them, 192.168.1.1 corresponds to the format <hostname>; 10000 corresponds to the format <port>; Gateway corresponds to the format <space>; / nc / data / hello.nc corresponds to the format <path>;.nc corresponds to the format <type>;pm25[0:1:443][0:1:381] corresponds to the format <constraints>This request means: remotely access the hello.nc file in the directory / nc / data on HDFS and extract all data in the range [0,441][0,381] in the pm25 variable.

[0066] S202: The DAP converter converts the data access request into a query instruction, and sends the query instruction to the BES client;

[0067] Figure 3 This is a schematic diagram of the DAP converter provided in the embodiment of this application. Please refer to Figure 3 In a possible implementation, the DAP converter converts the data access request into a query instruction including:

[0068] The data access request is parsed using a URL parser to obtain several parameters in the data access request. A query instruction is generated based on the parameters according to the specified grammatical rules. The query instruction includes a query instruction fragment corresponding to each of the several parameters. The generated query instruction is in XML format.

[0069] For example, converting a specific request http: / / 192.168.1.1:10000 / gateway / nc / data / hell o.nc.nc?pm25[0:1:443][0:1:381] into a query instruction, the generated query instruction part format is as follows:

[0070]

[0071] Among them, gateway corresponds to space="gateway" in the instruction;

[0072] / nc / data / hello.nc corresponds to the resourceID in the instruction;

[0073] .nc corresponds to the returnType in the instruction;

[0074] pm25[1:0:443][1:0:381] corresponds to the constraintExpression in the instruction.

[0075] The above command means: remotely access the hello.nc file in the directory / nc / data on HDFS, extract all data in the range [0,441][0,381] from the pm25 variable, and return it in the format of a netcdf file.

[0076] S203: The BES client sends a query instruction to the BES server; the BES server obtains the query result on HDFS according to the query instruction and sends the query result to the BES client.

[0077] The BES client sends a query command in XML format to the BES server. The BES server first accesses the hello.nc file through Gateway based on the resourceID in the query command, then extracts the data to be queried based on the constraintExpression, converts the extracted data into the data format .nc corresponding to the returnType, generates the query results, and returns the query results to the BES client.

[0078] Specifically, the BES server obtains the query results from HDFS according to the query instructions. This application can access and obtain files on HDFS through the following HTTP request format:

[0079] http: / / <hostname> : <port> / <operation> / <path> ？ <params>

[0080] Among them, hostname represents the host path of HDFS, port represents the port of HDFS, operarion represents the access operation, and the parameters include download (representing obtaining data) and upload (representing uploading data). Path represents the access path of atmospheric environment data on HDFS, and params represents the parameters required to perform the operation.

[0081] The Gateway interface processes this request. It includes Routers, Services, and the HDFS client. Routers distribute requests, Services parse parameters and send them to the HDFS client, and the HDFS client receives and responds to the parameters, returning the response to the BES server.

[0082] In a possible implementation, the BES client sending a query instruction to the BES server includes:

[0083] S2031: The BES client establishes a point-to-point transmission protocol with the BES server.

[0084] The BES server is an independently running C / S architecture service program. After running, it waits to receive query instructions sent by the BES client, queries data according to the query instructions, and returns the query results to the BES client.

[0085] In a more specific embodiment, the BES client and the BES server establish a point-to-point transmission protocol including:

[0086] S20311: The BES client sends a Transmission Control Protocol connection request to the BES server.

[0087] S20312: The BES server accepts the request and a transmission control protocol connection is established.

[0088] In a more specific embodiment, the BES server uses the SSL security protocol and encrypted transmission. The BES server then requires the BES client to provide verification information. If the verification information passes, a Transmission Control Protocol (TCP) connection is established. If the verification information connection times out or fails, the BES client exits, and the TCP connection establishment fails.

[0089] S20313: The BES client sends a connection request to the BES server to request the establishment of a point-to-point transmission protocol. The connection request includes an identifier representing the requested point-to-point protocol connection. Table 1 shows the information and description of the identifier.

[0090] Table 1

[0091]

[0092]

[0093] S20314: The BES server receives the connection request and returns an identifier to the BES client to confirm the request result.

[0094] For example, a BES client sends a request containing the identifier "PPTCLIENT_TESTING_CONNE CTION" to the BES server, requesting to establish a PPT protocol connection. After receiving the request, the BES server returns an identifier to indicate the request result based on its own status. If the returned identifier is "PPTSERVER_CONNECTION_OK," it means the BES server has accepted the request and can proceed with PPT protocol transmission. If the returned identifier is "PPT_PROTOCOL_UNDEFINED," it means the BES server is busy or the protocol is unavailable. If the returned identifier is "PPTSERVER_AUTHENTICATE," it means the BES server has configured the SSL security protocol and needs to enter the verification phase.

[0095] S2032: The BES client sends the query instruction to the BES server via the point-to-point transmission protocol.

[0096] Specifically, the BES client transmits the query instruction in XML format to the BES server through the PPT protocol.

[0097] S204: The BES client performs protocol conversion processing on the query result to obtain a processing result; and sends the processing result to the user client.

[0098] For example, the BES client converts the data returned by the BES server into the data format .nc corresponding to the returnType to generate a query result, and then returns the query result to the user client.

[0099] This application combines the OPeNDAP protocol and HDFS storage through the two interfaces of dap and gateway. This can not only give full play to the advantages of the OPeNDAP protocol and efficiently and conveniently transmit atmospheric environmental data, but also enjoy the management advantages of HDFS such as data redundancy backup.

[0100] Figure 4 This is a schematic diagram of the system provided in the embodiment of this application. Figure 4 As shown, in one possible embodiment, the data transmission system of the present application includes:

[0101] User client 401, used to send a data access request to the DAP converter;

[0102] A data access protocol converter 402 is used to convert a data access request into a query instruction and send the query instruction to a backend program client 403;

[0103] In some possible embodiments, converting a data access request into a query instruction includes:

[0104] The data access request is parsed to obtain several parameters in the data access request; a query instruction is generated according to the several parameters and in accordance with prescribed grammatical rules; the query instruction includes a query instruction fragment corresponding to each of the several parameters.

[0105] The backend program client 403 is used to send query instructions to the backend program server 404;

[0106] The backend program server 404 is used to obtain query results according to the query instruction and send the query results to the backend program client 403;

[0107] In some possible embodiments, the backend program client 403 sends a query instruction to the backend program server 404; the backend program server 404 obtains a query result according to the query instruction, and sends the query result to the backend program client 403, including:

[0108] The backend program client 403 establishes a point-to-point transmission protocol with the backend program server 404;

[0109] In some possible embodiments, the backend program client 403 sends a transmission control protocol connection request to the backend program server 404;

[0110] The backend program server 404 accepts the request and the Transmission Control Protocol connection is established;

[0111] The backend program client 403 sends a connection request to the backend program server 404 to apply for establishing a point-to-point transmission protocol; wherein the connection request includes an identifier representing the requested point-to-point protocol connection;

[0112] The backend program server 404 receives the connection request and returns an identifier to the backend program client 403 to confirm the request result.

[0113] The backend program client 403 sends the query instruction to the backend program server 404 via the point-to-point transmission protocol;

[0114] The backend program server 404 obtains the query result on the distributed file system 405 according to the query instruction and sends the query result to the backend program client 403;

[0115] The backend program client 403 receives and processes the query result.

[0116] The backend program client 403 is further configured to perform protocol conversion processing on the query result to obtain a processing result; and send the processing result to the user client 401 .

[0117] Figure 5 The schematic diagram of the computer device structure provided by the embodiment of this specification is shown. The computer device may include: a processor 510, a memory 520, an input / output interface 530, a communication interface 540 and a bus 550. The processor 510, the memory 520, the input / output interface 530 and the communication interface 540 are connected to each other through the bus 550. The computer device can be used to perform the above Figure 2 The method shown.

[0118] The processor 510 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0119] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 520 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 520 and is called and executed by the processor 510.

[0120] The input / output interface 530 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0121] The communication interface 540 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0122] The bus 550 comprises a pathway for transmitting information between the various components of the device (eg, the processor 510 , the memory 520 , the input / output interface 530 , and the communication interface 540 ).

[0123] It should be noted that although the above device only shows the processor 510, the memory 520, the input / output interface 530, the communication interface 540, and the bus 550, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0124] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0125] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0126] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / params> < / path> < / operation> < / port> < / hostname> < / constraints> < / type> < / path> < / space> < / port> < / hostname> < / constraints> < / type> < / path> < / space> < / port> < / hostname>

Claims

1. A method for transmitting atmospheric science data based on HDFS, characterized in that: The transmission method includes: The user client sends a data access request to the data access protocol converter; The data access protocol converter converts the data access request into a query instruction and sends the query instruction to the back-end program client; The backend program client sends a query instruction to the backend program server; the backend program server obtains the query result in the distributed file system according to the query instruction and sends the query result to the backend program client; The backend client performs protocol conversion on the query results to obtain the processing results; Send the processing results to the user client; Among them, the backend program client and the backend program server establish a point-to-point transmission protocol, including: The backend program client sends a transmission control protocol connection request to the backend program server; the backend program server accepts the request and the transmission control protocol connection is established; the backend program client sends a connection request to the backend program server to apply for establishing a point-to-point transmission protocol; wherein the connection request includes an identifier representing the requested point-to-point protocol connection; the backend program server receives the connection request and returns the identifier to the backend program client to confirm the request result; The data is atmospheric environment data. The OPeNDAP protocol and HDFS are combined through the dap and gateway interfaces to query the data stored on HDFS.

2. The atmospheric science data transmission method based on HDFS according to claim 1, characterized in that: The data access protocol converter converts data access requests into query instructions, including: The data access request is parsed to obtain several parameters in the data access request; a query instruction is generated according to the several parameters and in accordance with prescribed grammatical rules; the query instruction includes a query instruction fragment corresponding to each of the several parameters.

3. The atmospheric science data transmission method based on HDFS according to claim 1, characterized in that: The backend client sends a query command to the backend server. The backend server obtains the query results from the distributed file system according to the query command and sends the query results to the backend client, including: The backend program client sends the query instruction to the backend program server through the point-to-point transmission protocol; The backend program server obtains the query results according to the query instructions and sends the query results to the backend program client; The backend client receives and processes the query results.

4. The atmospheric science data transmission method based on HDFS according to claim 1, characterized in that: The data access request is a URL request, and the query instruction is in XML format.

5. An atmospheric science data transmission system based on HDFS, characterized in that: The transmission system comprises: A user client, used to send a data access request to the data access protocol converter; A data access protocol converter is used to convert data access requests into query instructions and send the query instructions to the back-end program client; The backend program client is used to send query instructions to the backend program server; The backend program server is used to obtain query results according to the query instructions and send the query results to the backend program client; The backend program client is also used to perform protocol conversion processing on the query results to obtain the processing results; and send the processing results to the user client; Among them, the backend program client and the backend program server establish a point-to-point transmission protocol, including: The backend program client sends a transmission control protocol connection request to the backend program server; the backend program server accepts the request and the transmission control protocol connection is established; the backend program client sends a connection request to the server to request to establish a point-to-point transmission protocol; wherein the connection request includes an identifier representing the requested point-to-point protocol connection; the backend program server receives the connection request and returns the identifier to the backend program client to confirm the request result; The data is atmospheric environment data. The OPeNDAP protocol and HDFS are combined through the dap and gateway interfaces to query the data stored on HDFS.

6. The atmospheric science data transmission system based on HDFS according to claim 5, characterized in that: Convert data access requests into query instructions, including: The data access request is parsed to obtain several parameters in the data access request; a query instruction is generated according to the several parameters and in accordance with prescribed grammatical rules; the query instruction includes a query instruction fragment corresponding to each of the several parameters.

7. The atmospheric science data transmission system based on HDFS according to claim 5, characterized in that: The backend program client sends a query instruction to the backend program server; the backend program server obtains the query result according to the query instruction and sends the query result to the backend program client, including: The backend program client sends the query instruction to the backend program server through the point-to-point transmission protocol; The backend program server obtains the query results according to the query instructions and sends the query results to the backend program client; The backend client receives and processes the query results.

8. The HDFS-based atmospheric science data transmission system according to claim 5, characterized in that: The data access request is a URL request, and the query instruction is in XML format.

Images