Astronomical data transmission method and device

Abstract

The present application relates to an astronomical data transmission method and device, the method comprises: splitting the astronomical data file to be transmitted into a plurality of split files, and generating split description information, the split description information includes the total number of split files, the identification and serial number of each split file; calling the first production end, encapsulating the split description information into split description information messages, encapsulating each split file into each split file message, and sequentially sending the split description information messages and each split file message to the second device; receiving the feedback message through the second service end and storing the feedback message into the second message queue; consuming the feedback message from the second message queue through the second consumption end; when the feedback message is the reverse confirmation message, according to the missing split file information in the reverse confirmation message, calling the first production end, re-encapsulating the missing split file into a split file message and sending it to the second device.

Description

Technical Field

[0001] This invention relates to the field of astronomical data transmission technology, and more specifically to an astronomical data transmission method and apparatus. Background Technology

[0002] In large-scale scientific data transmission scenarios (such as multi-site archiving of radio telescope observation data and data synchronization of distributed storage systems), it is necessary to distribute massive amounts of observation data from the data source publisher to multiple remote receiving nodes at high speed and reliability to complete multi-site data archiving and collaborative processing. This type of astronomical data transmission scenario has stringent business characteristics: First, the data volume is enormous, with a single astronomical observation file reaching several gigabytes to tens of gigabytes, requiring batch processing of massive amounts of file data in a single transmission task; second, high transmission throughput requirements necessitate continuous high-speed data transmission relying on limited network resources to ensure real-time archiving of observation data; third, stringent transmission reliability standards require avoiding network fluctuations, node process failures, and other issues to prevent data loss, corruption, and distortion; fourth, support for multi-destination distribution is required, with the same set of astronomical observation data needing to be simultaneously pushed to multiple remote distributed receiving nodes; fifth, the transmission network environment is complex, with multi-node transmission across wide area networks experiencing high latency and dynamic fluctuations in network status.

[0003] Traditional message middleware transmission architectures are ill-suited to the needs of this scenario. Conventional small file direct transmission modes are limited by the length of a single message and cannot handle the transmission of extremely large files. General large file transmission solutions lack targeted fragmentation scheduling, integrity verification, and multi-node synchronization mechanisms, which can easily lead to problems such as fragment loss, inconsistent archiving, and low throughput utilization, making it difficult to meet the high-speed, high-reliability, and multi-site synchronous transmission requirements of astronomical observation data.

[0004] Therefore, there is an urgent need for an efficient and stable data transmission method that is adapted to the characteristics of astronomical big data. Summary of the Invention

[0005] The purpose of this invention is to provide an astronomical data transmission method and apparatus to achieve efficient and stable transmission of astronomical data.

[0006] To achieve the above objectives, the present invention provides an astronomical data transmission method applied to an astronomical data transmission system. The astronomical data transmission system includes a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The method is applied to the first device and includes:

[0007] The astronomical data file to be transmitted is split into multiple fragment files, and fragment description information is generated. The fragment description information includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0008] The first production terminal is invoked to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and sequentially send the fragment description information messages and fragment file messages to the second device; so that the second device receives and consumes the fragment description information messages and fragment file messages through the first server and the first consumer terminal, and generates a feedback message when a preset condition is met, and then invokes the second production terminal to send the feedback message to the first device; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a reverse confirmation message indicating that there are missing fragment files and carrying information about the missing fragment files;

[0009] The second server receives the feedback message and stores the feedback message in the second message queue.

[0010] The feedback message is consumed from the second message queue through the second consumer terminal;

[0011] When the feedback message is the reverse confirmation message, based on the missing fragment file information in the reverse confirmation message, the first production end is invoked to repackage the missing fragment file into a fragment file message and send it to the second device.

[0012] Optionally, it also includes:

[0013] Record the sending time of the last fragment file message;

[0014] Starting from the time the last fragment file message was sent, if the feedback message is not received within a preset time, the first production end is invoked to resend the fragment description information message and each fragment file message to the second device.

[0015] Another aspect of the present invention provides an astronomical data transmission method applied to an astronomical data transmission system. The astronomical data transmission system includes a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The method is applied to the second device and includes:

[0016] The first server receives the fragmentation description information message and each fragment file message sequentially sent by the first device to the first production terminal, and stores the fragmentation description information message and each fragment file message into the first message queue; wherein the fragmentation description information message is obtained by encapsulating fragmentation description information, each fragment file message is obtained by encapsulating each fragment file separately, each fragment file is obtained by splitting the astronomical data file to be transmitted, and the fragmentation description information includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0017] The first consumer consumes messages from the first message queue.

[0018] When preset conditions are met, a feedback message is generated; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files.

[0019] The second production end is invoked to send the feedback message to the first device; so that the first device receives the feedback message through the second server, and when the feedback message is a reverse confirmation message, the first production end is invoked to resend the missing fragment file message to the second device;

[0020] When the feedback message is a reverse confirmation message, the first server and the first consumer re-receive and consume the missing fragment file message sent by the first device calling the first producer, and regenerate the feedback message;

[0021] When the feedback message is a positive confirmation message, all archived fragment files are reassembled to obtain the reassembled file.

[0022] Optionally, consuming messages in the first message queue specifically includes:

[0023] Retrieve fragment description information messages and fragment file messages from the first message queue;

[0024] Obtain fragment description information based on the fragment description information message, and archive the fragment description information;

[0025] For each fragment file message, retrieve the fragment file based on the fragment file message and archive the fragment file.

[0026] Optionally, a feedback message is generated, specifically including:

[0027] The integrity of each archived fragment file is checked based on the archived fragment description information to determine whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of missing fragment files are obtained as missing fragment file information, and a reverse confirmation message is generated based on the missing fragment file information.

[0028] In another aspect, the present invention provides an astronomical data transmission device applied to an astronomical data transmission system. The astronomical data transmission system includes a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The device is applied to the first device and includes:

[0029] The splitting module is used to split the astronomical data file to be transmitted into multiple fragment files and generate fragment description information, which includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0030] The first sending module is configured to call the first production terminal to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and sequentially send the fragment description information messages and fragment file messages to the second device; so that the second device receives and consumes the fragment description information messages and fragment file messages through the first server and the first consumer terminal, and generates a feedback message when a preset condition is met, and then calls the second production terminal to send the feedback message to the first device; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a reverse confirmation message indicating that there are missing fragment files and carrying information about the missing fragment files;

[0031] The first receiving module is used to receive the feedback message through the second server and store the feedback message into the second message queue;

[0032] The first consumption module is used to consume the feedback message from the second message queue through the second consumer terminal;

[0033] The missing file retransmission module is used to, when the feedback message is the reverse confirmation message, call the first production end according to the missing file information in the reverse confirmation message, repackage the missing file into a file message and send it to the second device.

[0034] Optionally, it also includes a timeout retransmission module, wherein the timeout retransmission module is specifically configured as follows:

[0035] Record the sending time of the last fragment file message;

[0036] Starting from the time the last fragment file message was sent, if the feedback message is not received within a preset time, the first production end is invoked to resend the fragment description information message and each fragment file message to the second device.

[0037] In another aspect, the present invention provides an astronomical data transmission device applied to an astronomical data transmission system. The astronomical data transmission system includes a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The device is applied to the second device and includes:

[0038] The second receiving module is used to receive, through the first server, the fragment description information message and each fragment file message sent sequentially by the first device calling the first production terminal, and store the fragment description information message and each fragment file message into the first message queue; wherein the fragment description information message is obtained by encapsulating fragment description information, each fragment file message is obtained by encapsulating each fragment file separately, each fragment file is obtained by splitting the astronomical data file to be transmitted, and the fragment description information includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0039] The second consumption module is used to consume messages in the first message queue through the first consumer terminal;

[0040] The generation module is used to generate a feedback message when preset conditions are met; wherein the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files.

[0041] The second sending module is used to call the second production end to send the feedback message to the first device; so that the first device receives the feedback message through the second server, and when the feedback message is a reverse confirmation message, calls the first production end to resend the missing fragment file message to the second device;

[0042] The re-receive module is used to re-receive and consume the missing fragment file message sent by the first device calling the first producer when the feedback message is a reverse confirmation message, and regenerate the feedback message.

[0043] The reassembly module is used to reassemble all archived fragment files when the feedback message is a positive confirmation message, so as to obtain the reassembled file.

[0044] Optionally, consuming messages in the first message queue specifically includes:

[0045] Retrieve fragment description information messages and fragment file messages from the first message queue;

[0046] Obtain fragment description information based on the fragment description information message, and archive the fragment description information;

[0047] For each fragment file message, retrieve the fragment file based on the fragment file message and archive the fragment file.

[0048] Optionally, a feedback message is generated, specifically including:

[0049] The integrity of each archived fragment file is checked based on the archived fragment description information to determine whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of missing fragment files are obtained as missing fragment file information, and a reverse confirmation message is generated based on the missing fragment file information. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of an astronomical data transmission system according to an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of an astronomical data transmission method applied to a first device according to an embodiment of the present invention;

[0052] Figure 3 This is a flowchart of an astronomical data transmission method applied to a second device according to an embodiment of the present invention;

[0053] Figure 4 This is a structural block diagram of an astronomical data transmission device applied to a first device according to an embodiment of the present invention;

[0054] Figure 5 This is a structural block diagram of an astronomical data transmission device applied to a second device according to an embodiment of the present invention. Detailed Implementation

[0055] The preferred embodiments of the present invention are given below with reference to the accompanying drawings and described in detail.

[0056] This invention provides an astronomical data transmission method, which is applied to an astronomical data transmission system, such as... Figure 1 As shown, the astronomical data transmission system includes a first device 10, a second device 20, a first message middleware, and a second message middleware. The first message middleware includes a first producer 31, a first server 32, and a first consumer 33. The second message middleware includes a second producer 41, a second server 42, and a second consumer 43. The first producer 31 is deployed on the first device 10, the first server 32 and the first consumer 33 are deployed on the second device 10, the second producer 41 is deployed on the second device 20, and the second server 42 and the second consumer 43 are deployed on the first device 10. The second device 20 is configured with a first message queue, and the first device 10 is configured with a second message queue. The first device 10 is the device for sending astronomical data, and the second device is the device for receiving astronomical data; that is, the astronomical data needs to be transmitted completely from the first device 10 to the second device 20.

[0057] like Figure 2 As shown, this embodiment of the invention provides an astronomical data transmission method, which is applied to a first device 10 and includes the following steps:

[0058] S110: Split the astronomical data file to be transmitted into multiple fragment files and generate fragment description information, which includes the total number of fragment files, the identifier and sequence number of each fragment file.

[0059] Because astronomical data files are massive, with single files reaching tens of gigabytes in size, they cannot be transmitted in a single complete transfer. Therefore, it is necessary to fragment the original files, that is, to split the astronomical data files to be transmitted into multiple fragment files, and then transmit each fragment file in batches to improve transmission stability and feasibility. The size of each fragment file can be flexibly set as needed. After fragmentation, fragment description information can be generated, including the total number of fragment files, the identifier and sequence number of each fragment file. The fragment description information is used by the second device 20 to complete fragment verification and file reassembly.

[0060] S120: The first producer 31 is invoked to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and send the fragment description information messages and fragment file messages to the second device 20 in sequence, so that the second device 20 receives and consumes the fragment description information messages and fragment file messages through the first server 32 and the first consumer 33, generates a feedback message based on the consumption result, and then invokes the second producer 41 to send the feedback message to the first device 10; wherein, the feedback message is a positive acknowledgment message (ACK) indicating that all fragment files have been archived or a negative acknowledgment message (NACK) indicating that there are missing fragment files and carrying information about the missing fragment files.

[0061] After the fragmentation process is completed, the first device 10 can call the first producer 31 to sequentially encapsulate the fragmentation description information and each fragment file into messages. Each message includes a message header and a message body. The message header contains metadata information used for message routing and distribution, transmission control, link tracing, and integrity verification. The message body is the payload of the message, used to carry business data. After each message is encapsulated, the first producer 31 sends the message to the first server 32 of the second device 20. The first server 32 is responsible for receiving all transmitted messages and storing each message in the first message queue according to the order of receipt. When a message is successfully stored in the first message queue, the first consumer 33 can pull and consume the corresponding message from the first message queue. The first producer 31, the first server 32, and the first consumer 33 rely on message middleware to realize message production, storage, and consumption. Their basic interaction logic is conventional technology in this field and will not be described in detail here.

[0062] The first consumer terminal 33 consumes messages from the first message queue, specifically including: obtaining fragment description information messages and fragment file messages from the first message queue; sequentially performing parsing and validity verification on the obtained fragment description information messages; after completing the verification and parsing operations, extracting the corresponding fragment description information from the fragment description information messages; persistently storing the extracted fragment description information to the local disk and archiving it; and sequentially performing parsing and validity verification on each fragment file message; after completing the verification and parsing operations, extracting the corresponding fragment file from the fragment file message; persistently storing the extracted fragment file to the local disk and archiving it.

[0063] When the preset conditions are met, the second device 20 can perform integrity verification on each archived fragment file according to the archived fragment description information, thereby determining whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of the fragment files that have not been archived (i.e., missing fragment files) are obtained as missing fragment file information, and then a reverse confirmation message is generated based on the missing fragment file information. During integrity verification, the archived fragment description information in the disk can be read to parse the total number of fragment files corresponding to this transmission task and the set of identifiers for all fragment files. Then, the file archive storage directory can be traversed, all archived fragment files in the current directory can be scanned, the identifier of each archived fragment file can be extracted, and a local set of archived fragment file identifiers can be generated. Bidirectional consistency verification is then performed: 1) The actual number of archived fragments in the local directory is compared with the total number of fragments recorded in the fragment description information; 2) The identifier of each fragment file in the set of identifiers for all fragment files is traversed, and the existence of each fragment identifier in the local set of archived fragment identifiers is checked one by one. If the total number of fragment files is consistent, and all fragment file identifiers can be matched with corresponding fragment files in the local set of archived fragment file identifiers, then the archived fragment files are considered complete, that is, all fragment files have been archived. If the number of fragment files is inconsistent, and there is a case where the identifier of any fragment file does not match the corresponding archived fragment file, then the fragment file is considered incomplete, and the number and identifier of missing files can be obtained as missing file information. The preset conditions can be that the first consumer 33 has completed archiving the last fragment file, or that a preset time has elapsed since the first message was received, or any other suitable condition.

[0064] After generating the feedback message, the second device 20 will call the second production terminal 41 to send it to the second server 42 of the first device 10.

[0065] S130: Receive feedback messages through the second server 42 and store the feedback messages in the second message queue.

[0066] The first device 10 can receive feedback messages through the second server 42 and store them in the second message queue.

[0067] S140: Consume feedback messages from the second message queue via the second consumer 43.

[0068] After the second server 42 stores the feedback message in the second message queue, the second consumer 43 can consume the feedback message from the second message queue based on the preset mechanism within the second message middleware. Consuming the feedback message includes parsing it to extract the information it carries, thereby determining the type of the feedback message.

[0069] The second production end 41, the second server end 42, and the second consumer end 43 rely on message middleware to realize message production, storage, and consumption. Their basic interaction logic is a conventional technology in this field and will not be explained in detail here.

[0070] S150: When the feedback message is a reverse confirmation message, based on the missing fragment file information in the reverse confirmation message, the first production end 31 is called to repackage the missing fragment file into a fragment file message and send it to the second device 20.

[0071] The first device 10 can determine the message type by the information carried in the feedback message. When the feedback message is a reverse confirmation message, the first device 10 can call the first producer 31 based on the missing fragment file information in the reverse confirmation message to repackage the missing fragment file into a fragment file message and resend it to the second device 20. The second device 20 re-receives and consumes the resent fragment file message through the first server 32 and the first consumer 33, re-archives it, re-verifies the integrity of each archived fragment file, and generates a feedback message. The above process continues until the first device 10 receives a positive confirmation message. After the first device 10 receives a positive confirmation message, it indicates that the astronomical data file to be transmitted has been completely transmitted to the second device 20, and the transmission is complete.

[0072] The first and second message middleware can be any suitable existing message middleware. Since the first message middleware is used to transmit fragmented files, while the second message middleware is only used to transmit feedback messages, the first message middleware can be a large file message middleware, and the second message middleware can be a small file middleware, thus improving data transmission efficiency. For example, the first message middleware can be ZeroMQ, and the second message middleware can be RabbitMQ.

[0073] In some embodiments, the astronomical data transmission method may further include:

[0074] While the first device 10 calls the first production terminal 31 to send the last fragment file message, it also records the sending time of the last fragment file message.

[0075] Starting from the time the last fragment file message was sent, if no feedback message is received from the second device 20 within a preset time, the first production end is invoked to resend the fragment description information message and each fragment file message to the second device 20.

[0076] If no feedback message is received within the preset time, it indicates a problem with message transmission. The second device 20 has not received the fragment description message and the fragment file messages, therefore retransmission needs to be initiated. Specifically, the first device 10 can start a timer after the last fragment file message is sent. The timer duration is a preset time. When a feedback message is received, the timer is canceled. When no feedback message is received, the timer continues to run. When the timer expires, the first device 10 initiates retransmission.

[0077] like Figure 3 As shown, this embodiment of the invention also provides an astronomical data transmission method, which is applied to a second device 20 and includes the following steps:

[0078] S210: The first server 32 receives the fragmentation description information message and each fragment file message sent sequentially by the first device 10 calling the first production terminal 31, and stores the fragmentation description information message and each fragment file message into the first message queue.

[0079] As mentioned earlier, when a file needs to be transferred, the first device 10 splits the astronomical data file to be transferred into multiple fragment files and generates fragment description information. Then, it calls the first production terminal 31 to encapsulate the fragment description information and each fragment file into fragment description information messages and fragment file messages, and sends them to the second device 20 in sequence. The second device 20 can then receive the fragment description information messages and fragment file messages in sequence through the first server 32, and store each message in the first message queue in sequence.

[0080] S220: Consume messages in the first consumption queue through the first consumer terminal 33.

[0081] After the first server 32 stores the message in the first consumption queue, the first consumer 33 reads the message from it and performs operations such as parsing and validity verification on the message in sequence. Then, it extracts the corresponding data from the message, persists it to the local disk, and completes the archiving. Specifically, the first consumer 33 can extract the fragment description information from the fragment description information message and store the fragment description information to the local disk to complete the archiving; for each fragment file message, the first consumer 33 can extract the corresponding fragment file from it and store it to the local disk to complete the archiving.

[0082] S230: When the preset conditions are met, a feedback message is generated, wherein the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files.

[0083] In some embodiments, generating a feedback message specifically includes:

[0084] The integrity of each archived fragment file is checked based on the archived fragment description information to determine whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of missing fragment files are obtained as missing fragment file information, and a reverse confirmation message is generated based on the missing fragment file information.

[0085] During integrity verification, the archived fragment description information in the disk can be read to parse the total number of fragment files corresponding to this transmission task and the set of identifiers for all fragment files. Then, the file archive storage directory can be traversed, all archived fragment files in the current directory can be scanned, the identifier of each archived fragment file can be extracted, and a local set of archived fragment file identifiers can be generated. Bidirectional consistency verification is then performed: 1) The actual number of archived fragments in the local directory is compared with the total number of fragments recorded in the fragment description information; 2) The identifier of each fragment file in the set of identifiers for all fragment files is traversed, and the existence of each fragment identifier in the local set of archived fragment identifiers is checked one by one. If the total number of fragment files is consistent, and all fragment file identifiers can be matched with corresponding fragment files in the local set of archived fragment file identifiers, then the archived fragment files are considered complete, that is, all fragment files have been archived. If the number of fragment files is inconsistent, and there is a case where the identifier of any fragment file does not match the corresponding archived fragment file, then the fragment file is considered incomplete, and the number and identifier of missing files can be obtained as missing file information.

[0086] The preset conditions can be that the first consumer 33 has completed archiving the last fragment file, or that a preset time has elapsed since the first message was received, or any other suitable condition.

[0087] S240: Call the second production terminal 41 to send the feedback message to the first device 10; so that the first device 10 receives the feedback message through the second server 42, and when the feedback message is a reverse confirmation message, call the first production terminal 31 to resend the missing fragment file message to the second device 20.

[0088] After generating the feedback message, the second device 20 calls the second producer 41 to send the feedback message to the second server 42 of the first device 10. The first device 10 receives the feedback message through the second server 42, stores it in the second message queue, and consumes the feedback message through the second consumer 43. When the feedback message is a reverse confirmation message, the first device 10 calls the first producer 31 to repackage the missing fragment file into a missing fragment file message and resend it to the second device 20.

[0089] S250: When the feedback message is a reverse confirmation message, the first server 32 and the first consumer 33 re-receive and consume the missing fragment file message sent by the first device 10 calling the first producer 31, and regenerate the feedback message.

[0090] When the feedback message is a reverse confirmation message, the second device 20 will receive the missing fragment file message again through the first server 32 and store it back into the first message queue; the first consumer 33 will consume the missing fragment file message again, and the second device 20 will also re-verify the integrity of each archived fragment file and generate a feedback message; the above process continues until the feedback message is a positive confirmation message.

[0091] S260: When the feedback message is a positive confirmation message, reassemble all archived fragment files to obtain the reassembled file.

[0092] When the feedback message is a positive confirmation message, it means that all fragment files have been archived. At this time, the second device 20 can reassemble all the archived fragment files according to the fragment description information (for example, reassemble according to the sequence number of each fragment file in the fragment description information) to obtain the reassembled file. The reassembled file is consistent with the astronomical data file to be transmitted. In this way, the astronomical data file to be transmitted can be reliably transmitted from the first device 10 to the second device 20.

[0093] The astronomical data transmission method of this invention has the following advantages compared with the prior art:

[0094] 1. Significantly improved transmission speed: Astronomical data files are transmitted through the first message middleware, and feedback messages are transmitted through the second message middleware. That is, the data channel and the acknowledgment channel are completely decoupled. The sending thread can continuously push data without being blocked by waiting for feedback messages. In high-latency network (such as wide area network) environments, the transmission speed can be increased by several times or even an order of magnitude.

[0095] 2. Significantly enhanced transmission reliability: The second message queue adopts a persistent message queue, so feedback messages are not lost due to process restart, network interruption or server failure. The first device 10 can make retransmission decisions based on the accurate missing fragment files, avoiding the misjudgment of "not received" and large-scale duplicate retransmission caused by ACK loss in the traditional solution.

[0096] like Figure 4 As shown, this embodiment of the invention also provides an astronomical data transmission device, which is applied to a first device 10 and includes a splitting module 11, a first sending module 12, a first receiving module 13, a first consumption module 14, and a missing retransmission module 15.

[0097] The splitting module 11 is used to split the astronomical data file to be transmitted into multiple fragment files and generate fragment description information, which includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0098] The first sending module 12 is used to call the first production terminal 31 to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and sequentially send the fragment description information messages and fragment file messages to the second device 20; so that the second device 20 receives and consumes the fragment description information messages and fragment file messages through the first server terminal 32 and the first consumer terminal 33, and generates a feedback message when a preset condition is met, and then calls the second production terminal 41 to send the feedback message to the first device 10; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a reverse confirmation message indicating that there are missing fragment files and carrying information about the missing fragment files;

[0099] The first receiving module 13 is used to receive the feedback message through the second server 32 and store the feedback message into the second message queue;

[0100] The first consumption module 14 is used to consume the feedback message from the second message queue through the second consumption terminal 33;

[0101] The missing retransmission module 15 is used to, when the feedback message is the reverse confirmation message, call the first production terminal 31 according to the missing fragment file information in the reverse confirmation message, repackage the missing fragment file into a fragment file message and send it to the second device 20.

[0102] The astronomical data transmission device applied to the first device 10 may also include a timeout retransmission module, which is specifically configured to: record the sending time of the last fragment file message; starting from the sending time of the last fragment file message, if the feedback message is not received within a preset time, the first production terminal 31 is invoked to resend the fragment description information message and each fragment file message to the second device 20.

[0103] The splitting module 11, the first sending module 12, the first receiving module 13, the first consumption module 14, the missing retransmission module 15, and the timeout retransmission module are functional modules corresponding to steps S110-S160 of the method embodiment applied to the first device 10. Their specific implementation methods can be referred to the description in the method embodiment, and will not be repeated here.

[0104] like Figure 5 As shown, this embodiment of the invention also provides an astronomical data transmission device, which is applied to a second device and includes a second receiving module 21, a second consumption module 22, a generation module 23, a second sending module 24, a re-receiving module 25, and a reassembly module 26.

[0105] The second receiving module 21 is used to receive, through the first server, the fragment description information message and each fragment file message sent sequentially by the first device calling the first production terminal, and store the fragment description information message and each fragment file message into the first message queue; wherein the fragment description information message is obtained by encapsulating fragment description information, each fragment file message is obtained by encapsulating each fragment file separately, each fragment file is obtained by splitting the astronomical data file to be transmitted, and the fragment description information includes the total number of fragment files, the identifier and sequence number of each fragment file;

[0106] The second consumption module 22 is used to consume messages in the first message queue through the first consumer terminal;

[0107] The generation module 23 is used to generate a feedback message when a preset condition is met; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files.

[0108] The second sending module 24 is used to call the second production end to send the feedback message to the first device; so that the first device receives the feedback message through the second server, and when the feedback message is a reverse confirmation message, calls the first production end to resend the missing fragment file message to the second device;

[0109] The re-receiving module 25 is used to re-receive and consume the missing fragment file message sent by the first device calling the first producer when the feedback message is a reverse confirmation message, and regenerate the feedback message.

[0110] The reassembly module 26 is used to reassemble all archived fragment files when the feedback message is a positive confirmation message, so as to obtain the reassembled file.

[0111] The second receiving module 21, the second consuming module 22, the generating module 23, the second sending module 24, the re-receiving module 25, and the reassembly module 26 are functional modules corresponding to steps S210-S260 in the method embodiment applied to the second device 20. Their specific implementation methods can be found in the description in the method embodiment, and will not be repeated here.

[0112] The astronomical data transmission device of this invention transmits astronomical data files through a first message middleware and feedback messages through a second message middleware. That is, the data channel and the acknowledgment channel are completely decoupled. The sending thread can continuously push data without being blocked by waiting for feedback messages. In high-latency network (such as wide area network) environments, the transmission speed can be increased by several times or even an order of magnitude. The second message queue adopts a persistent message queue, so the feedback message is not lost due to process restart, network interruption or server failure. The first device 10 can make retransmission decisions based on accurate missing fragment files, avoiding the misjudgment of "not received" and large-scale repeated retransmission caused by ACK loss in traditional solutions.

[0113] The astronomical data described in this application may include raw observation data streams directly acquired by various astronomical observation devices in space and ground-based observation scenarios, or structured observation data obtained after preprocessing, format conversion, noise reduction and correction, feature extraction, pixel mapping, and time-series encoding of the raw observation data. This type of astronomical data may be at least one of the following: image data, spectral data, time-series sampling data, position and attitude measurement data, radiation intensity sampling data, detector array readout data, ephemeris positioning data, observation condition parameter data, observation metadata, and equipment status synchronization data in any observation frequency band such as optical band, infrared band, radio band, X-ray band, and high-energy particle band. The astronomical data may be encapsulated in any transmission format such as binary, serialized message, file fragmentation, and streaming data packet, and has the characteristics of large data volume, continuous streaming output, strong temporal correlation, observation batch binding, and multi-channel synchronous acquisition. Reliable and orderly transmission, integrity verification, and data archiving and retention are required between observation equipment, data storage nodes, and back-end processing equipment.

[0114] The first device 10 and the second device 20 in this application are any electronic data processing devices capable of data transmission and reception, network communication, data caching, persistent data storage, and data verification processing. The first device 10 can be an observation-side data transmission device such as an astronomical observation detector, an astronomical telescope terminal, an array detection and acquisition device, a spaceborne observation payload, a ground-based front-end acquisition and control device, an observation data sharding and sending server, or an observation site edge computing node. The second device 20 can be a receiving-side data processing device such as an astronomical data archiving server, an observation data processing workstation, a distributed storage cluster, an astronomical data center back-end processing device, a remote data backup node, a data verification and management terminal, or an observation task scheduling service device. The first device 10 and the second device 20 can establish a data transmission channel based on any communication method such as wired Ethernet, fiber optic communication, wireless microwave communication, satellite communication, or a dedicated telemetry and control link.

[0115] Another embodiment of this specification provides a readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the steps of the astronomical data transmission method described in the above embodiments of this specification.

[0116] Another embodiment of this specification provides an electronic device, which includes a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it performs the steps of the astronomical data transmission method in the above embodiments of this specification.

[0117] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0118] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0119] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by electronic devices. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0129] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0130] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. That is, all simple and equivalent changes and modifications made based on the claims and description of this invention fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A method for transmitting astronomical data, characterized in that, An astronomical data transmission system is applied, comprising a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The method is applied to the first device and includes: The astronomical data file to be transmitted is split into multiple fragment files, and fragment description information is generated. The fragment description information includes the total number of fragment files, the identifier and sequence number of each fragment file; The first production terminal is invoked to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and sequentially send the fragment description information messages and fragment file messages to the second device; so that the second device receives and consumes the fragment description information messages and fragment file messages through the first server and the first consumer terminal, and generates a feedback message when a preset condition is met, and then invokes the second production terminal to send the feedback message to the first device; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a reverse confirmation message indicating that there are missing fragment files and carrying information about the missing fragment files; The second server receives the feedback message and stores the feedback message in the second message queue. The feedback message is consumed from the second message queue through the second consumer terminal; When the feedback message is the reverse confirmation message, based on the missing fragment file information in the reverse confirmation message, the first production end is invoked to repackage the missing fragment file into a fragment file message and send it to the second device.

2. The astronomical data transmission method according to claim 1, characterized in that, Also includes: Record the sending time of the last fragment file message; Starting from the time the last fragment file message was sent, if the feedback message is not received within a preset time, the first production end is invoked to resend the fragment description information message and each fragment file message to the second device.

3. A method for transmitting astronomical data, characterized in that, An astronomical data transmission system is applied, comprising a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The method is applied to the second device and includes: The first server receives the fragmentation description information message and each fragment file message sequentially sent by the first device to the first production terminal, and stores the fragmentation description information message and each fragment file message into the first message queue; wherein the fragmentation description information message is obtained by encapsulating fragmentation description information, each fragment file message is obtained by encapsulating each fragment file separately, each fragment file is obtained by splitting the astronomical data file to be transmitted, and the fragmentation description information includes the total number of fragment files, the identifier and sequence number of each fragment file; The first consumer consumes messages from the first message queue. When preset conditions are met, a feedback message is generated; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files. The second production end is invoked to send the feedback message to the first device; so that the first device receives the feedback message through the second server, and when the feedback message is a reverse confirmation message, the first production end is invoked to resend the missing fragment file message to the second device; When the feedback message is a reverse confirmation message, the first server and the first consumer re-receive and consume the missing fragment file message sent by the first device calling the first producer, and regenerate the feedback message; When the feedback message is a positive confirmation message, all archived fragment files are reassembled to obtain the reassembled file.

4. The astronomical data transmission method according to claim 3, characterized in that, Consuming messages in the first message queue specifically includes: Retrieve fragment description information messages and fragment file messages from the first message queue; Obtain fragment description information based on the fragment description information message, and archive the fragment description information; For each fragment file message, retrieve the fragment file based on the fragment file message and archive the fragment file.

5. The astronomical data transmission method according to claim 4, characterized in that, Generate feedback messages, specifically including: The integrity of each archived fragment file is checked based on the archived fragment description information to determine whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of missing fragment files are obtained as missing fragment file information, and a reverse confirmation message is generated based on the missing fragment file information.

6. An astronomical data transmission device, characterized in that, An astronomical data transmission system is applied, comprising a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The device is applied to the first device and includes: The splitting module is used to split the astronomical data file to be transmitted into multiple fragment files and generate fragment description information, which includes the total number of fragment files, the identifier and sequence number of each fragment file; The first sending module is configured to call the first production terminal to encapsulate the fragment description information into fragment description information messages, encapsulate each fragment file into fragment file messages, and sequentially send the fragment description information messages and fragment file messages to the second device; so that the second device receives and consumes the fragment description information messages and fragment file messages through the first server and the first consumer terminal, and generates a feedback message when a preset condition is met, and then calls the second production terminal to send the feedback message to the first device; wherein, the feedback message is a positive confirmation message indicating that all fragment files have been archived or a reverse confirmation message indicating that there are missing fragment files and carrying information about the missing fragment files; The first receiving module is used to receive the feedback message through the second server and store the feedback message into the second message queue; The first consumption module is used to consume the feedback message from the second message queue through the second consumer terminal; The missing file retransmission module is used to, when the feedback message is the reverse confirmation message, call the first production end according to the missing file information in the reverse confirmation message, repackage the missing file into a file message and send it to the second device.

7. The astronomical data transmission device according to claim 6, characterized in that, It also includes a timeout retransmission module, which is specifically configured as follows: Record the sending time of the last fragment file message; Starting from the time the last fragment file message was sent, if the feedback message is not received within a preset time, the first production end is invoked to resend the fragment description information message and each fragment file message to the second device.

8. An astronomical data transmission device, characterized in that, An astronomical data transmission system is applied, comprising a first device, a second device, a first message middleware, and a second message middleware. The first message middleware includes a first producer, a first server, and a first consumer. The second message middleware includes a second producer, a second server, and a second consumer. The first producer is deployed on the first device, and the first server and the first consumer are deployed on the second device. The second producer is deployed on the second device, and the second server and the second consumer are deployed on the first device. The device is applied to the second device and includes: The second receiving module is used to receive, through the first server, the fragment description information message and each fragment file message sent sequentially by the first device calling the first production terminal, and store the fragment description information message and each fragment file message into the first message queue; wherein the fragment description information message is obtained by encapsulating fragment description information, each fragment file message is obtained by encapsulating each fragment file separately, each fragment file is obtained by splitting the astronomical data file to be transmitted, and the fragment description information includes the total number of fragment files, the identifier and sequence number of each fragment file; The second consumption module is used to consume messages in the first message queue through the first consumer terminal; The generation module is used to generate a feedback message when preset conditions are met; wherein the feedback message is a positive confirmation message indicating that all fragment files have been archived or a negative confirmation message indicating that there are missing fragment files that have not been archived and carrying information about the missing fragment files. The second sending module is used to call the second production end to send the feedback message to the first device; so that the first device receives the feedback message through the second server, and when the feedback message is a reverse confirmation message, calls the first production end to resend the missing fragment file message to the second device; The re-receive module is used to re-receive and consume the missing fragment file message sent by the first device calling the first producer when the feedback message is a reverse confirmation message, through the first server and the first consumer, and regenerate the feedback message. The reassembly module is used to reassemble all archived fragment files when the feedback message is a positive confirmation message, so as to obtain the reassembled file.

9. The astronomical data transmission device according to claim 8, characterized in that, Consuming messages in the first message queue specifically includes: Retrieve fragment description information messages and fragment file messages from the first message queue; Obtain fragment description information based on the fragment description information message, and archive the fragment description information; For each fragment file message, retrieve the fragment file based on the fragment file message and archive the fragment file.

10. The astronomical data transmission device according to claim 9, characterized in that, Generate feedback messages, specifically including: The integrity of each archived fragment file is checked based on the archived fragment description information to determine whether all fragment files recorded in the fragment description information have been archived. If so, a positive confirmation message is generated; if not, the number and identifier of missing fragment files are obtained as missing fragment file information, and a reverse confirmation message is generated based on the missing fragment file information.

Images