A box delivery service system and method

Abstract

This invention discloses a box delivery service system and method. The system includes a box service network, a box cluster, and a ZooKeeper service module. The box service network has multiple server nodes, and the box cluster contains multiple box entities. Each box entity is deployed independently and connected to the box service network. The box service network establishes a connection with a work order system wirelessly to achieve data interaction. The ZooKeeper service module is used to manage the status of the box service network. The steps include connecting the box to the Internet; configuring the box-related information and initializing the hard drive; automatically registering the box to the box service network after connecting to the Internet; and when data is generated, the box service network notifies the box to start a data download task.

Description

Technical Field

[0001] This invention relates to the field of box delivery services, and more specifically to a box delivery service system and method. Background Technology

[0002] Second-generation high-throughput gene sequencing technology can process up to 3.6Tbx2 (Nova seq 6000) per sequencing run. Sequencing companies typically store the sequencing data on public clouds (Alibaba, AWS, Tencent, etc.) for customers to download themselves. Without a dedicated sequencing platform, customers need a dedicated computer with the appropriate download tools for the public cloud and a high-bandwidth network connection. Manual monitoring is required to confirm data generation and add the download process only after data generation is complete. If the download is interrupted due to network issues or other reasons, the service must be restarted manually, consuming significant manpower. Furthermore, data integrity cannot be guaranteed. Due to the large volume of data, downloading takes a long time, and any other reason during this period can lead to data transmission interruptions, incomplete files, and errors in subsequent bioinformatics processing. Therefore, current sequencing data delivery methods suffer from drawbacks such as massive data volumes, complex manual download management, and long processing times. How to provide dedicated software systems for high-speed delivery of sequencing data to various research institutions, universities, hospitals, medical institutions, pharmaceutical companies, and gene companies, and solve the problem of the last mile of gene data delivery, is a technical challenge that urgently needs to be solved in this field.

[0003] Currently, there is no product on the market that can solve the problem of high-speed delivery of massive amounts of genetic data. This invention aims to solve this technical problem by providing a box delivery service system. After adopting this box, the system inside the box intelligently senses the generation of new data in the cloud, automatically synchronizes multi-threaded fragment downloads in real time, and has a data retransmission mechanism. This solves the shortcomings of sequencing data, such as the complexity and time-consuming manual download and management due to the large amount of data. Data delivery is fast, timely, complete, and secure, and supports mainstream domestic public cloud object storage services (and can be dynamically expanded to support more vendors as needed). After the data is transmitted to the FlashTest Magic Box, it can be accessed through the user's network as network storage (NAS), and also supports copying and exporting data via high-speed USB 3.0 external hard drives. Summary of the Invention

[0004] This invention provides a box delivery service system and a box delivery service method. The box includes a complete solution of software, hardware, and network. Customers only need to pay to enjoy high-speed data download to their local devices. It automatically detects cloud data generation without manual intervention, and the sequencing data is automatically downloaded, or even generated and downloaded simultaneously, greatly accelerating data delivery time. It is convenient and simple to use, saves investment, and eliminates the need for computers and dedicated download software, thus overcoming the shortcomings of existing technologies.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solutions:

[0006] Currently, the design and development process of the kit needs to consider two types of users: the end customers of sequencing services. The main challenge for these customers is that the kit operates in an unmaintained environment. The kit's service must be robust enough to cope with various possible scenarios and impose certain requirements on remote maintenance.

[0007] For labs located close to clients, the lab PM needs to insert the client's storage devices and complete data output for multiple clients. This scenario requires clear identification of each batch of data to avoid outputting incorrect data to the client.

[0008] Firstly, a box delivery service system includes a box service network, a box cluster, and a ZooKeeper service module. The ZooKeeper service module has a built-in open-source distributed state management software. The box service network has multiple server nodes, and the box cluster contains multiple box entities. Each box entity is deployed independently and connected to the box service network. The box service network establishes a connection with a work order system wirelessly to achieve data interaction. The ZooKeeper service module is used to manage the state of the box service network. When a box entity needs to communicate with the work order system, it does not connect to any particular work order system, but rather connects to the box service network. The work order system then connects to the box service network to perform data transmission / box management and other tasks.

[0009] In the aforementioned box delivery service system, the server node is a CubeMeshNode (box network node), and the box itself is a CubeNode (box node).

[0010] The aforementioned box delivery service system includes a server node comprising a Remote Procedure Call service module (RPCServer), a cluster status management module (ClusterKeeper), a first metadata module (MetaData), a box runtime information module (RuntimeCubeInfo), a box status management module (CubeKeeper), a box status heartbeat detection module (CubeKeepAliveWorker), a box connection resolution module (CubeConnectionResolver), and a box message communication protocol library module (CubeTalk Message Library).

[0011] The Remote Procedure Call Service (RPCServer) module serves as the entry point for the work order system to remotely access the box.

[0012] The cluster state management module (ClusterKeeper) is used to maintain the state data between the server node and the ZooKeeper service module, and to register the server node to the box service network;

[0013] The first metadata module (MetaData) stores the metadata library required by the box service network;

[0014] The RuntimeCubeInfo module is used to abstract the status information of an online cube.

[0015] The CubeKeeper module is responsible for maintaining the connection status with a Cube.

[0016] The box status heartbeat detection module (CubeKeepAliveWorker) is responsible for polling the boxes at low frequency to avoid zombie boxes caused by inactive connections;

[0017] The CubeConnectionResolver module acts as the server for box communication, responsible for establishing connections with the boxes and encoding / decoding communication.

[0018] The CubeTalk Message Library module is a protocol library for CubeTalk communication, which contains all the message types that CubeTalk may use.

[0019] The box is used to receive data pushes from the work order system, manage the storage devices connected to the box, and output data to the storage devices. The box includes a metadata storage module, a network node connection module, a volume space usage check thread module, a disk operation thread module, a disk status monitoring module, a data download and release module, a second metadata module, a large data reading module, a download module, a hard disk initialization module, and a box communication protocol module.

[0020] The metadata storage module is used to manage the metadata structure of local storage devices;

[0021] The network node connection module (MeshConnector) is the client for box communication, responsible for establishing and encoding / decoding communication with the box service network;

[0022] The Volume Space Usage Check Thread module is responsible for monitoring changes in the remaining capacity of available partitions.

[0023] The disk operation thread module (DiskManipulatingThread) is responsible for managing the local disk, including mounting, ejecting, and initializing.

[0024] The disk status monitoring module (DiskWatcher) is responsible for monitoring changes to the storage device, such as the insertion of a new disk;

[0025] The data download and release module (DownloadExtractor) is responsible for outputting a completed or partially completed download to an external storage device;

[0026] The second metadata module stores metadata structures related to the business.

[0027] The BLOBReader module provides the ability to read downloaded data. To control the number of file handles downloaded, the data from a single download is integrated into a single blob. The BLOBReader module is required to correctly read the downloaded content.

[0028] The download module (Downloader) is a downloader used to perform the download action for a given file. The download module (Downloader) itself does not handle the connection data transmission or file writing; instead, it controls three components to complete the download:

[0029] For files without content, after obtaining the HTTPS download address, the file is split into several data blocks. The download module (Downloader) creates multiple fragment downloaders and downloads these data blocks in parallel through a local queue.

[0030] The FragCommitter write will write the completed frag to the local disk according to the offset;

[0031] ContentDownloader - For files containing content, the content is retrieved directly from the box service network;

[0032] The disk initialization module (FSMakerchusi) provides the ability to initialize disks and offers multiple implementations to support different types of file partition tables and partition formats.

[0033] The CubeTalk Protocol module stores the network protocol used for communication between the box and the box service network.

[0034] In the aforementioned box delivery service system, the communication protocol between the remote procedure call service module (RPCServer) and the work order system is the Thrift protocol.

[0035] The first metadata module (MetaData) stores metadata including CubeInfo, HDDInfo, Download, and Extraction information. CubeInfo is the registration information of the Cube, HDDInfo is an encapsulation of a known hard disk, Download is an encapsulation of a download task, and Extraction is an encapsulation of process data output to external storage.

[0036] The metadata structure within the Storage Metadata module includes BlockInfo, DiskInfo, VolumeInfo, DiskFlag, and InternalStorageDisk. BlockInfo abstracts all Linux block devices, including those that are not readily identifiable. DiskInfo abstracts a storage device. VolumeInfo abstracts a partition on a disk. DiskFlag abstracts a partition flag on a disk partition table. InternalStorageDisk abstracts and marks a DiskInfo, serving as internal storage. Internal storage can be used to receive data pushes from the ticket system, while other disks can be used to output the pushed data.

[0037] The metadata structure within the second metadata module includes Download, DownloadEntry, Frag, EntryState (download state), DownloadTaskState, Extraction, and ExtractionEntry. Download represents an abstraction of a single data push; DownloadEntry represents an abstraction of a downloaded file (which can be either containing or not); Frag represents a downloaded file without content; and Frag abstracts a segmented DownloadEntry; EntryState represents the state of a DownloadEntry during the download process; DownloadTaskState represents the state of a download; Extraction represents an abstraction of a data output process; and ExtractionEntry represents an abstraction of a file being extracted during a data output process.

[0038] In the aforementioned box delivery service system, the network protocol includes a negotiation phase and a communication phase;

[0039] The negotiation phase is used to establish a link between the box and the service network. Whether it is an uplink or downlink, the process is the same. The negotiation phase process is as follows:

[0040] Step A1: The box establishes a TCP (Transmission Control Protocol) connection with the box service network and sends the box's protocol version to the box service network;

[0041] Step A2: The box service network replies with the protocol version to the box;

[0042] Step A3: Is the box response protocol version compatible?

[0043] If incompatible, the box service network will disconnect;

[0044] If compatible, the box service network generates an RSA key and sends the RSA public key length and RSA public key from the RSA key to the box;

[0045] Step A4: The box generates an identifier containing the box's own device ID and connection type. Then, it encrypts the identifier using the read RSA public key and sends the RSA public key length and the identifier RSA key to the box's service network.

[0046] Step A5: After the box service network decrypts the identifier using the RSA private key in the RSA key, it checks the device registration information in the identifier and sends the identification result to the box;

[0047] Step A6: If the result is not a complete recognition, the box will disconnect from the network;

[0048] Step A7: The box service network sends a negotiation completion identifier;

[0049] Step A8: Enter the communication phase;

[0050] After the negotiation phase is completed, the link will enter this phase, which is a question-and-answer format that loops infinitely. During this phase, the communication behavior between the box and the service network is consistent. The process of this communication phase is as follows:

[0051] Step B1: Read 4 bytes from the connection between the box and the box service process;

[0052] Step B2: After identifying the 4 bytes read, obtain the message length M;

[0053] Step B3: Read M bytes of data from the connection;

[0054] Step B4: Decode the M-byte data using AES. The key used for decoding by the service network is the key used when the box registers with the service network. The key used for decoding by the box itself is the key configured on the local machine. For the same box, the keys of the two must match.

[0055] Step B5: Decode and process the received message;

[0056] Step B6: Repeat steps B1-B5 for the next loop.

[0057] When receiving data pushed by the work order system, the box needs to go through the following stages:

[0058] Creating a download task: The work order system creates a download for the box through the box's service network RPC service. The box cannot see the original OSS key of each file, but it can know the ID and file name of each file.

[0059] Get Download Address: The box sends a request to the service network to obtain the address of a file. The box uses the file ID to identify the file to be obtained. The service network finds the file requested by the box and generates an HTTPS download address without access secret through Alibaba Cloud's object signature function.

[0060] Data Segmentation: After obtaining the download URL, the box uses an HTTP range request to segment the data into several smaller chunks;

[0061] Download and assemble: The box uses the built-in FragDownloader to download the split data fragments separately, and then hands them over to the Frag Comitter to merge them and write them into the final BLOB. When all fragments have been downloaded, the download of a single file is considered complete.

[0062] Secondly, a box delivery service method includes the following steps:

[0063] Step C1: Connect the box to the internet;

[0064] Step C2: Configure the box-related information and initialize the hard drive;

[0065] Step C3: After the box connects to the internet, it will automatically register with the box service network;

[0066] Step C4: When data is generated, the box service network notifies the box to start the data download task.

[0067] In the aforementioned box delivery service method, step C1 requires configuring PPPoE, password, internal network card parameters, and the box's communication public key after connecting the box to the Internet.

[0068] In step C2, use your mobile phone to scan the device QR code on the box shell to enter the configuration interface and configure the relevant information of the box.

[0069] In step C3, after the box connects to the network, it automatically registers with the box service network according to the internal preset configuration.

[0070] The aforementioned box delivery service method further includes step C5: After the data download is complete, it can be copied to the place where it is needed via an external hard drive or accessed via the network to the directory where the box's data is located.

[0071] The relevant information of the box also includes operator-related information, and step C2 also includes binding the operator to the box.

[0072] Thirdly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the second aspects.

[0073] The technical solution provided by the box delivery service system and method of the present invention, as described above, has the following technical effects:

[0074] After adopting this box, the internal system intelligently senses the generation of new data in the cloud, automatically synchronizes multi-threaded fragment download in real time, and has a data retransmission mechanism. This solves the shortcomings of sequencing data, such as the large amount of data, the complexity and time-consuming manual download and management. Data delivery is fast, timely, complete, and secure, and it supports mainstream domestic public cloud object storage services (and can be dynamically expanded to support more vendors as needed). After the data is transmitted to the FlashTest Magic Box, it can be accessed through the user's network as a network storage device (NAS). It also supports copying and exporting data via a high-speed USB 3.0 external hard drive.

[0075] The box includes a complete solution encompassing software, hardware, and network, allowing customers to enjoy high-speed data downloads to their local devices simply by paying.

[0076] Automatic detection of cloud-based data generation requires no manual intervention, and sequencing data is automatically downloaded, allowing for simultaneous generation and download, which greatly accelerates data delivery time.

[0077] It is easy and simple to use, saves investment, and no longer requires a computer or dedicated download software;

[0078] For customers with poor network conditions (lack of bandwidth) or who cannot access the internet for security reasons, a fast data delivery method that eliminates the need to mail hard drives is provided.

[0079] Supports copying data from external USB devices;

[0080] It supports local network data service (NFSCIFS) and can provide file services over a local area network;

[0081] It features robust data storage security and supports RAID 015 and similar architectures. Attached Figure Description

[0082] Figure 1 This is a schematic diagram of the structure of a box delivery service system according to the present invention;

[0083] Figure 2 Network node diagram for the box service;

[0084] Figure 3 For box node diagrams;

[0085] Figure 4 Diagram showing communication negotiation between the box;

[0086] Figure 5 Here is a diagram of the communication message structure of the box;

[0087] Figure 6 This is a schematic diagram of data download stage 1;

[0088] Figure 7 This is a schematic diagram of data download stage 2;

[0089] Figure 8 This is a schematic diagram of data download stage 3;

[0090] Figure 9 This is a schematic diagram of data download stage 4;

[0091] Figure 10 This is a flowchart of a box delivery service method according to the present invention.

[0092] The accompanying figure is labeled as follows:

[0093] Box Service Network 100, Box Cluster 200, ZooKeeper Service Module 300, Work Order System 400, Server Node 110, Box Body 210, Remote Procedure Call Service Module 111, Cluster Status Management Module 112, First Metadata Module 113, Box Runtime Information Module 114, Box Status Management Module 115, Box Status Heartbeat Detection Module 116, Box Connection Parsing Module 117, Box Message Communication Protocol Library Module 118, Metadata Storage Module 211, Network Node Connection Module 212, Volume Space Usage Detection Thread Module 213, Disk Operation Thread Module 214, Disk Status Monitoring Module 215, Data Download and Release Module 216, Second Metadata Module 217, Large Block Data Reading Module 218, Download Module 219, Hard Disk Initialization Module 220, Box Communication Protocol Module 221. Detailed Implementation

[0094] To make the technical means, inventive features, objectives and effects of the invention easier to understand, the invention will be further explained below with reference to specific illustrations.

[0095] The box serves as a temporary storage for data delivered by the work order system to the end user. The box must meet the following requirements: it must be able to receive and store data generated by the work order system, including but not limited to: FastQ (a sequencing format file), BCL (a raw sequencing data format file generated by a sequencer), and UnknownFastQ (FastQ file for data whose attribution cannot be determined); it must be able to connect to an external storage device and copy the received data to that device; and it must provide network access, allowing direct access to the received data within the box from the local area network where the box is located.

[0096] It should have management capabilities for external storage devices, with functionality covering the entire process: storage device insertion → data copying → storage device removal. The enclosure itself should also have monitoring capabilities, able to monitor the following:

[0097] The box itself can monitor the data transmission process from the work order system, the data copying process to external storage devices, and provides a user interface for execution.

[0098] The management box stores data received from the work order system, manages external storage devices, executes the process of outputting data to external storage devices, manages the network services built into the box, and ensures that communication between the box and the work order system is secure and controlled.

[0099] In the application process, there are scenarios where the box actively connects to the work order system, and there are also scenarios where the work order system actively connects to the box. However, since the box is deployed in an uncontrolled network environment, this means that each box does not have a fixed network entry point (IP address / domain name, etc.).

[0100] Therefore, the box needs to have passive communication capabilities with the work order system, meaning that once the box establishes a connection with the work order system, the work order system can send requests to the box at any time.

[0101] The first embodiment of the present invention provides a box delivery service system. The purpose is that the box contains a complete solution of software, hardware and network, and customers only need to pay to enjoy the service of high-speed data download to their local devices; it automatically detects cloud data generation without manual intervention, and the sequencing data is automatically downloaded, and can also be generated and downloaded at the same time, which greatly speeds up the data delivery time; it is convenient and simple to use, saves investment, and eliminates the need to equip computers and install special download software for downloading.

[0102] like Figure 1 As shown, in the first aspect, a box delivery service system includes a box service network 100, a box cluster 200, and a ZooKeeper service module 300. The ZooKeeper service module 300 has a built-in open-source distributed state management software. The box service network 100 has multiple server nodes 110. The box cluster 200 includes multiple box bodies 210. Each box body 210 is deployed separately and connected to the box service network 100. The box service network 100 establishes a connection with a work order system 400 wirelessly to achieve data interaction. The ZooKeeper service module 300 is used to manage the state of the box service network 100. When a box body 210 needs to communicate with the work order system 400, it does not connect to a specific work order system 400, but connects to the box service network 100. The work order system 400 then connects to the box service network 100 to perform data transmission / box management and other tasks.

[0103] In the aforementioned box delivery service system, server node 110 is a CubeMeshNode (box network node), and box body 210 is a CubeNode (box node).

[0104] like Figure 2-3 As shown, the aforementioned box delivery service system includes a server node 110 comprising a remote procedure call service module 111 (RPCServer), a cluster status management module 112 (ClusterKeeper), a first metadata module 113 (MetaData), a box runtime information module 114 (RuntimeCubeInfo), a box status management module 115 (CubeKeeper), a box status heartbeat detection module 116 (CubeKeepAliveWorker), a box connection resolution module 117 (CubeConnectionResolver), and a box message communication protocol library module 118 (CubeTalk Message Library).

[0105] The Remote Procedure Call Service module 111 (RPCServer) serves as the entry point for the 400 remote access box of the work order system;

[0106] The cluster status management module 112 (ClusterKeeper) is used to maintain the status data between the server node 110 and the ZooKeeper service module 300, and to register the server node 110 to the box service network 100.

[0107] The first metadata module 113 (MetaData) stores the metadata library that the box service network 100 needs to use;

[0108] The RuntimeCubeInfo module 114 is used to abstract the status information of an online cube.

[0109] The CubeKeeper module 115 is responsible for maintaining the connection status with a Cube.

[0110] The box status heartbeat detection module 116 (CubeKeepAliveWorker) is responsible for polling the box at low frequency to avoid zombie boxes caused by inactive connections;

[0111] The CubeConnectionResolver module 117 acts as the server for box communication, responsible for establishing connections with the boxes and encoding / decoding communication.

[0112] The CubeTalk Message Library module 118 is the protocol library for CubeTalk communication, which contains all the message types that CubeTalk may use.

[0113] The box is used to receive data pushes from the work order system 400, manage the storage devices connected to the box, and output data to the storage devices. The box includes a metadata storage module 211 (Storage Metadata), a network node connection module 212 (MeshConnector), a volume space usage detection thread module 213 (VolumeSpaceUsageCheckThread), a disk operation thread module 214 (DiskManipulatingThread), a disk status monitoring module 215 (DiskWatcher), a data download and release module 216 (DownloadExtractor), a second metadata module 217 (Metadata), a large data reading module 218 (BLOBReader), a download module 219 (Downloader), a hard disk initialization module 220 (FSMakerchusi), and a box communication protocol module 221 (Cube TalkProtocol).

[0114] The metadata storage module 211 (Storage Metadata) is used to manage the metadata structure of local storage devices;

[0115] The network node connection module 212 (MeshConnector) is the client for box communication, responsible for establishing communication and encoding / decoding with the box service network 100;

[0116] The Volume Space Usage Check Thread module 213 is responsible for monitoring changes in the remaining capacity of available partitions;

[0117] The disk operation thread module 214 (DiskManipulatingThread) is responsible for handling the management of the local disk, including mounting, ejecting, and initialization.

[0118] The disk status monitoring module 215 (DiskWatcher) is responsible for monitoring changes to the storage device, such as the insertion of a new disk;

[0119] The data download and release module 216 (DownloadExtractor) is responsible for outputting a completed or partially completed download to an external storage device;

[0120] The second metadata module 217 (Metadata) stores the metadata structure related to the business;

[0121] The large data reading module 218 (BLOBReader) is used to provide the ability to read downloaded data. In order to control the number of file handles downloaded, the data downloaded in a single download is integrated into a blob. The large data reading module 218 (BLOBReader) is required to correctly read the downloaded content.

[0122] Download module 219 (Downloader) is a downloader used to perform the download action for a given file. Download module 219 (Downloader) itself does not handle the connection data transmission or file writing; instead, it controls three components to complete the download:

[0123] For files without content, after obtaining the HTTPS download address, the file will be split into several data blocks. Download module 219 (Downloader) will create multiple FragDownloaders and download these data blocks in parallel through a local queue.

[0124] The FragCommitter write will write the completed frag to the local disk according to the offset;

[0125] Content Downloader - For files containing content, the content is retrieved directly from the box service network 100;

[0126] The hard disk initialization module 220 (FSMakerchusi) provides the ability to initialize the disk and offers multiple implementations to support different types of file partition tables and partition formats;

[0127] The Box Communication Protocol Module 221 (CubeTalk Protocol) stores the network protocol used for communication between the box and the Box Service Network 100.

[0128] In the aforementioned box delivery service system, the communication protocol between the remote procedure call service module 111 (RPCServer) and the work order system 400 is the Thrift protocol.

[0129] The first metadata module 113 (MetaData) stores metadata including box information (CubeInfo), hard disk information (HDDInfo), download information (Download), and extraction information (Extraction). Box information (CubeInfo) is the registration information of the box, hard disk information (HDDInfo) encapsulates a known hard disk, download information (Download) encapsulates a download task, and extraction information encapsulates the process data of outputting data to external storage.

[0130] The metadata structure within the Storage Metadata module includes BlockInfo, DiskInfo, VolumeInfo, DiskFlag, and InternalStorageDisk. BlockInfo abstracts all Linux block devices, including those that are not yet identifiable. DiskInfo abstracts a storage device. VolumeInfo abstracts a partition on a disk. DiskFlag abstracts a partition flag on the disk partition table. InternalStorageDisk is used to abstract and tag a DiskInfo, serving as internal storage. Internal storage can be used to receive data pushes from the 400 ticketing system, while other disks can be used to output the pushed data.

[0131] The metadata structure within the second metadata module 217 (Metadata) includes Download, Download Entry, Frag, EntryState (Download State), Download Task State, Extraction, and Extraction Entry. Download represents an abstraction of a single data push, Download Entry represents an abstraction of a downloaded file (which can be either containing or not containing content), Frag represents a downloaded file without content, and Frag abstracts a segmented Download Entry, EntryState represents the state of a Download Entry during the download process, Download Task State represents the state of a download, Extraction represents the abstraction of a data output process, and Extraction Entry represents an abstraction of a file that is extracted during a data output process.

[0132] In the aforementioned box delivery service system, the network protocol includes a negotiation phase and a communication phase;

[0133] like Figure 4 As shown, the negotiation phase is used to establish a link between the box and the service network. Whether it is an uplink or downlink, the process is the same. The negotiation phase process is as follows:

[0134] Step A1: The box establishes a TCP (Transmission Control Protocol) connection with the box service network 100 and sends the box's protocol version (2 bytes unsigned int) to the box service network 100;

[0135] Step A2: The box service network 100 replies with the protocol version (2 bytes unsigned int) to the box;

[0136] Step A3: Check if the box response protocol version is compatible, 1 = compatible, 2 = incompatible (2 bytes unsigned int);

[0137] If incompatible, the box service network 100 will disconnect;

[0138] If compatible, the box service network 100 generates an RSA key and sends the RSA public key length n (2 bytes unsigned int) and the RSA public key (n bytes) to the box;

[0139] Step A4: The box generates an identifier containing its own device ID and connection type. Then, it encrypts the identifier using the read RSA public key and sends the RSA public key length (2 bytes unsigned int) and the identifier to the box's service network 100. The identifier contains the box's own device ID and connection type (uplink / downlink). Then, it encrypts the identifier using the read RSA public key and sends the identifier length (2 bytes unsigned int) and the identifier.

[0140] Step A5: After the box service network 100 decrypts the identifier using the RSA private key in the RSA key, it checks the device registration information in the identifier and sends the identification result to the box. Identification result: 1 = Identification completed, 2 = Box cannot be identified, 3 = Box is disabled;

[0141] Step A6: If the result is not a complete recognition, the box will disconnect from the network;

[0142] Step A7: The box service network 100 sends a negotiation completion identifier 1 (1 unsigned byte);

[0143] Step A8: Enter the communication phase;

[0144] like Figure 5 As shown, after the negotiation phase is completed, the link will enter this phase, which is a question-and-answer format that loops infinitely. During this phase, the communication behavior between the box and the service network is the same. The communication phase process is as follows:

[0145] Step B1: Read 4 bytes (4 bytes unsigned int) from the connection between the box and the box service process;

[0146] Step B2: After identifying the 4 bytes read, obtain the message length M;

[0147] Step B3: Read M bytes of data from the connection;

[0148] Step B4: Decode the M-byte data using AES. The key used for decoding by the service network is the key used when the box registers with the service network. The key used for decoding by the box itself is the key configured on the local machine. For the same box, the keys of the two must match.

[0149] Step B5: Decode and process the received message;

[0150] Step B6: Repeat steps B1-B5 for the next loop.

[0151] When receiving data pushed by the work order system's 400 hotline, the box needs to go through the following stages:

[0152] like Figure 6 As shown, to create a download task: the work order system 400 creates a Download to the box through the RPC service of the box service network 100. The box cannot see the original OSS key of each file, but it can know the ID and file name of each file.

[0153] like Figure 7 As shown, to obtain the download address: The box sends a request to the service network to obtain the address of a file. The box uses the file ID to identify the file to be obtained. The service network finds the file requested by the box and generates an HTTPS download address without access secret through Alibaba Cloud's object signature function.

[0154] like Figure 8 As shown, the data is segmented: after the box obtains the download URL, the HTTP range request divides the data into several small blocks;

[0155] like Figure 9 As shown, the download and assembly process is as follows: The box uses the built-in FragDownloader to download the segmented data fragments separately, and then hands them over to the Frag Comitter to merge them and write them into the final BLOB. When all fragments have been downloaded, the download of a single file is considered complete.

[0156] like Figure 10 As shown, in a second aspect, a box delivery service method includes the following steps:

[0157] Step C1: Connect the box to the internet;

[0158] Step C2: Configure the box-related information and initialize the hard drive;

[0159] Step C3: After the box connects to the internet, it will automatically register with the box service network 100;

[0160] Step C4: When data is generated, the box service network 100 notifies the box to start the data download task.

[0161] In the aforementioned box delivery service method, step C1 requires configuring PPPoE, password, internal network card parameters, and the box's communication public key after connecting the box to the Internet.

[0162] In step C2, use your mobile phone to scan the device QR code on the box shell to enter the configuration interface and configure the relevant information of the box.

[0163] In step C3, after the box connects to the network, it automatically registers with the box service network 100 according to the internal preset configuration.

[0164] The aforementioned box delivery service method further includes step C5: After the data download is complete, it can be copied to the place where it is needed via an external hard drive or accessed via the network to the directory where the box's data is located.

[0165] The information related to the box also includes operator information, and step C2 also includes binding the operator to the box.

[0166] In summary, the present invention provides a box delivery service system and method. The box includes a complete solution of software, hardware, and network. Customers only need to pay to enjoy high-speed data download to their local devices. It automatically detects cloud data generation without manual intervention, and the sequencing data is automatically downloaded, or even generated and downloaded simultaneously, greatly accelerating data delivery time. It is convenient and simple to use, saves investment, and eliminates the need for computers and dedicated download software.

[0167] The specific embodiments of the invention have been described above. It should be understood that the invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood to be implemented in a manner common to the art; those skilled in the art can make various modifications or alterations within the scope of the claims, and make several simple deductions, variations or substitutions, which do not affect the substantive content of the invention.

Claims

1. A box delivery service system, characterized in that, The system includes a box service network, a box cluster, and a ZooKeeper service module. The box service network has multiple server nodes, and the box cluster contains multiple box entities. Each box entity is deployed independently and connected to the box service network. The box service network establishes a wireless connection with a work order system to achieve data interaction. The data received by the box entity is a gene sequencing file, including at least one of the following: FastQ sequencing format file, BCL raw sequencing data format file, or UnknownFastQ file. When receiving data pushed by the work order system, the box goes through the following stages: The work order system creates a download for the box through the box service network's RPC service. The box cannot see the original OSS key of each file, but it knows the ID and file name of each file. Get Download Address: The box sends a request to the service network to obtain the address of a file. The box uses the file ID to identify the file to be obtained. The service network finds the file requested by the box and generates an HTTPS download address without access secret through Alibaba Cloud's object signature function. Data Segmentation: After obtaining the download URL, the box uses an HTTP range request to segment the data into several smaller chunks; Download and assemble: The box uses the built-in FragDownloader to download the segmented data fragments separately, and then hands them over to the Frag Comitter to merge them and write them into the final BLOB. When all fragments have been downloaded, the download of a single file is considered complete. Network communication includes a negotiation phase and a communication phase. After ensuring the security of identity authentication through RSA, it switches to AES for efficient communication. The ZooKeeper service module is used to manage the status of the box service network. The box status heartbeat detection module polls the box at low frequency to avoid zombie boxes caused by inactive connections.

2. The box delivery service system as described in claim 1, characterized in that, The server node is CubeMeshNode, and the box itself is CubeNode.

3. The box delivery service system as described in claim 2, characterized in that, The server node includes a remote procedure call service module, a cluster status management module, a first metadata module, a box runtime information module, a box status management module, a box status heartbeat detection module, a box connection parsing module, and a box message communication protocol library module. The remote procedure call service module is the entry point for the work order system to remotely access the box; The cluster status management module is used to maintain the status data between the server node and the ZooKeeper service module, and to register the server node to the box service network; The first metadata module stores the metadata library required by the box service network; The box runtime information module is used to abstract the status information of an online box; The box status management module is responsible for maintaining the connection status with a box. The box connection parsing module acts as the server for box communication, responsible for establishing connections with the box and communication encoding and decoding. The box message communication protocol library is a protocol library for box communication, which contains all the message type data that the box may use. The box is used to receive data pushes from the work order system, manage the storage devices connected to the box, and output data to the storage devices. The box includes a metadata storage module, a network node connection module, a volume space utilization detection thread module, a disk operation thread module, a disk status monitoring module, a data download and release module, a second metadata module, a large block data reading module, a download module, a hard disk initialization module, and a box communication protocol module. The metadata storage module is used to manage the metadata structure of the local storage device; The network node connection module is the client for the box communication, and is responsible for establishing and encoding / decoding communication with the box service network; The volume space utilization detection thread module is responsible for monitoring changes in the remaining capacity of available partitions; The disk operation thread module is responsible for managing the local disk, including mounting, ejecting, and initializing. The disk status monitoring module is responsible for monitoring changes in the storage device; The data download and release module is responsible for outputting a completed or partially completed download to an external storage device; The second metadata module stores a metadata structure related to the business; The large data reading module is used to provide the ability to read downloaded data. The download module is a downloader used to perform a download action for a single Download. The hard disk initialization module is used to provide the ability to initialize the hard disk; The box communication protocol module stores the network protocol used for communication between the box and the box service network.

4. The box delivery service system as described in claim 3, characterized in that, The communication protocol between the remote procedure call service module and the work order system is the Thrift protocol. The metadata module stores metadata including box information, hard disk information, download information, and release information. The box information is the box's registration information, the hard disk information is a known hard disk, the download information is a download task, and the release information is process data for outputting data to external storage. The metadata structure within the metadata storage module includes block information, disk information, volume information, disk identifier, and internal storage disk. The block information abstracts all Linux block service devices, including those that are not identifiable. The disk information abstracts a storage device. The volume information abstracts a partition on a disk. The disk identifier is an abstract partition identifier on a disk partition table. The internal storage disk is used to abstract and mark a disk information, which is internal storage. The internal storage can be used to receive data pushes from the work order system, and the other disks can be used to output the pushed data. The metadata structure within the second metadata module includes download, download record, fragment, download status, download task status, release, and release status. Download is an abstraction of a data push; download record is an abstraction of a downloaded file; downloaded files are categorized as containing content or not containing content; fragment is a downloaded file without content; fragment abstracts a segment of the download record after it has been divided; download status abstracts the state of a download record during the download process; download task status abstracts the state of a download; release abstracts a data output process; and release status abstracts a file released during a data output process.

5. A box delivery service system as described in claim 4, characterized in that, The negotiation phase proceeds as follows: Step A1: The box establishes a Transmission Control Protocol (TCP) connection with the box service network and sends the box's protocol version to the box service network; Step A2: The box service network replies with the protocol version to the box; Step A3: Is the box response protocol version compatible? If incompatible, the box service network will disconnect; If compatible, the box service network generates an RSA key and sends the RSA public key length and RSA public key from the RSA key to the box; Step A4: The box generates an identifier containing the box's own device ID and connection type. Then, it encrypts the identifier using the read RSA public key and sends the RSA public key length and the identifier to the box's service network. Step A5: After the box service network decrypts the identifier using the RSA private key in the RSA key, it checks the device registration information in the identifier and sends the identification result to the box; Step A6: If the result is not a complete recognition, the box will disconnect from the network; Step A7: The box service network sends a negotiation completion identifier; Step A8: Enter the communication phase; The communication phase proceeds as follows: Step B1: Read 4 bytes from the connection between the box and the box service process; Step B2: After identifying the 4 bytes read, obtain the message length M; Step B3: Read M bytes of data from the connection; Step B4: Decode the M-length bytes of data using AES. The key used for decoding by the service network is the key used when the box registers with the service network. The key used for decoding by the box is the key configured on the local machine. For the same box, the keys of the two must match. Step B5: Decode and process the received messages; Step B6: Repeat steps B1-B5 for the next loop.

6. A box delivery service method based on the box delivery service system of claim 1, characterized in that, The data received by the box itself is a gene sequencing file, including at least one of the following: FastQ sequencing format file, BCL raw sequencing data format file, and UnknownFastQ file. The box status heartbeat detection module polls the box at low frequency to avoid zombie boxes caused by inactive connections. When receiving data pushed by the work order system, the box goes through the following stages: The work order system creates a download for the box through the box service network's RPC service. The box cannot see the original OSS key of each file, but it knows the ID and file name of each file. Get Download Address: The box sends a request to the service network to obtain the address of a file. The box uses the file ID to identify the file to be obtained. The service network finds the file requested by the box and generates an HTTPS download address without access secret through Alibaba Cloud's object signature function. Data Segmentation: After obtaining the download URL, the box uses an HTTP range request to segment the data into several smaller chunks; Download and assemble: The box uses the built-in FragDownloader to download the segmented data fragments separately, and then hands them over to the Frag Comitter to merge them and write them into the final BLOB. When all fragments have been downloaded, the download of a single file is considered complete. It also includes the following steps: Step C1: Connect the box to the internet; Step C2: Configure the box-related information and initialize the hard drive; Step C3: After the box connects to the internet, it will automatically register with the box service network; Step C4: When data is generated, the box service network notifies the box to start the data download task.

7. The box delivery service method as described in claim 6, characterized in that, After connecting the box to the Internet in step C1, you need to configure PPPoE, password, internal network card parameters, and the box's communication public key. In step C2, use your mobile phone to scan the device QR code on the box shell to enter the configuration interface and configure the relevant information of the box. In step C3, after the box connects to the network, it automatically registers with the box service network according to the internal preset configuration.

8. The box delivery service method as described in claim 7, characterized in that, It also includes step C5: After the data download is complete, it can be copied to the place where it is needed via an external hard drive or accessed via the network to the directory where the data of the box is located; The relevant information of the box also includes operator-related information, and step C2 also includes binding the operator to the box.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 6-8.

Images