Synchronization system, sending method and receiving method of internal and external network unstructured files

Abstract

The invention provides a synchronization system, a sending method and a receiving method of internal and external network unstructured files, and relates to the technical field of computer network data transmission, and the system comprises a sending end and a receiving end which communicate with each other through an internal and external network isolation device; the sending end is used for acquiring a to-be-synchronized unstructured file of the synchronization task; the to-be-synchronized unstructured file is partitioned and cut through a dynamic rule to generate a plurality of initial data blocks; adding corresponding metadata information to the initial data blocks, and generating a plurality of target metadata blocks based on the plurality of initial data blocks and the plurality of pieces of metadata information; the receiving end is used for acquiring a plurality of target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; and sorting and recombining the plurality of target metadata blocks to obtain a recombined and synchronized unstructured file of the synchronization task. According to the method, the transmission efficiency is improved through block transmission, the security of transmission data is guaranteed through the internal and external network isolation device by means of the target metadata block, and the transmission risk is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer network data transmission, in particular to a synchronization system, a sending method and a receiving method for unstructured files in an internal network and an external network. BACKGROUND

[0002] With the acceleration of enterprise digital transformation on the market, the proportion of unstructured data (such as videos, design drawings, log files) of the system has exceeded 80% of the total enterprise data, and a large amount of software and hardware storage services such as NAS (Network Attached Storage), OSS (Object Storage Service), and physical disks are invested. In particular, during the process of digital transformation, in order to strengthen and improve the security of the system and data, enterprises require the cross-network synchronization of unstructured data in an internal network and an external network isolation environment to surge. However, the traditional data synchronization scheme faces three core challenges: first, the traditional FTP / SFTP (File Transfer Protocol / SSH File Transfer Protocol) adopts a whole file transmission mode, and a single large video file transmission failure requires complete retransmission, resulting in low bandwidth utilization and low transmission efficiency, and there is no network isolation between transmission modes, which poses a security risk; second, in a logically isolated environment, manual intervention is required for copying through a USB flash disk (USB Flash Disk), which poses a risk of medium loss and virus spread, and lacks audit traceability, is inconvenient, has narrow applicability, and poses a security control conflict; third, the private network environment cannot be compared with the public cloud environment, and the system unstructured data cannot be transmitted and applied in real time, which easily causes data block loss, unstable synchronization service, and low synchronization task success rate.

[0003] Therefore, when synchronizing unstructured files in an internal network and an external network, the prior art has low transmission efficiency due to the use of a whole file transmission mode, and there is no network isolation between transmission modes, which poses a security risk. SUMMARY

[0004] To overcome the above shortcomings of the prior art, the present application provides a synchronization system for unstructured files in an internal network and an external network, comprising: a sending end and a receiving end communicating through an internal-external network isolation device;

[0005] The sending end is configured to obtain unstructured files to be synchronized for a synchronization task, cut and divide the unstructured files to be synchronized into a plurality of initial data blocks through dynamic rules, add corresponding metadata information to the initial data blocks, and generate a plurality of target metadata blocks based on the plurality of initial data blocks and the plurality of metadata information.

[0006] The receiving end is used for penetration transmission technology based on internal and external network isolation devices to obtain multiple target metadata blocks; the multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronous unstructured file of the synchronization task.

[0007] Preferably, when the sending end is located in an intranet environment and the receiving end is located in an extranet environment, the sending end is also used to: store multiple target metadata blocks in an intranet database;

[0008] The receiving end is specifically used for: based on the penetration transmission technology of the internal and external network isolation device, to periodically access the internal network database through polling or monitoring logs to obtain multiple target metadata blocks.

[0009] Preferably, when the sending end is located in an external network environment and the receiving end is located in an internal network environment, the receiving end is specifically used for:

[0010] Based on the penetration transmission technology of the internal and external network isolation device, multiple target metadata blocks are stored in the internal network database;

[0011] The internal database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

[0012] Preferably, the system also includes an application monitoring module located in an intranet environment;

[0013] The application monitoring module is specifically used to: query the process data generated during the execution of the synchronization task, and monitor the execution progress of the synchronization task based on the process data; the process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was synchronized successfully or not, and file logs.

[0014] Preferably, the application monitoring module is also used to: determine the peak and valley periods of business usage based on the number of successfully synchronized files within a preset period;

[0015] The sending end is also used to: dynamically adjust the block size of the unstructured file to be synchronized according to the peak and valley periods; and based on the adjusted block size, divide the unstructured file to be synchronized into blocks according to dynamic rules to generate multiple initial data blocks.

[0016] Preferably, the unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files;

[0017] The target metadata block includes a file information table and a file data table; the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time.

[0018] The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

[0019] Based on the same inventive concept, this invention also provides a method for sending unstructured files between internal and external networks, including:

[0020] Retrieve the unstructured files to be synchronized for the synchronization task;

[0021] For synchronizing unstructured files, multiple initial data blocks are generated by dynamically dividing and splitting them according to rules.

[0022] Add corresponding metadata information to the initial data block, generate multiple target metadata blocks based on multiple initial data blocks and multiple metadata information, so that the receiving end can obtain multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; sort and reorganize the multiple target metadata blocks to obtain the reorganized synchronous unstructured file of the synchronization task.

[0023] Based on the same inventive concept, the present invention also provides a device for sending unstructured files between internal and external networks, comprising:

[0024] The pending file acquisition module is used to acquire the unstructured files to be synchronized for the synchronization task;

[0025] The block splitting processing module is used to split unstructured files to be synchronized into multiple initial data blocks using dynamic rules.

[0026] The target metadata block generation module is used to add corresponding metadata information to the initial data block, and generate multiple target metadata blocks based on multiple initial data blocks and multiple metadata information. This enables the receiving end to obtain multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device. The multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronous unstructured file of the synchronization task.

[0027] Based on the same inventive concept, this invention also provides a method for receiving unstructured files on internal and external networks, including:

[0028] Based on the penetration transmission technology of the internal and external network isolation device, multiple target metadata blocks are obtained. The multiple target metadata blocks are the unstructured files to be synchronized by the sending end for the synchronization task. The unstructured files to be synchronized are divided into blocks by dynamic rules to generate multiple initial data blocks. Corresponding metadata information is added to the initial data blocks. The data is generated based on the multiple initial data blocks and the multiple metadata information.

[0029] Multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronized unstructured file for the synchronization task.

[0030] Based on the same inventive concept, the present invention also provides a device for receiving unstructured files on internal and external networks, comprising:

[0031] The target metadata block acquisition module is used to acquire multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device. The multiple target metadata blocks are the unstructured files to be synchronized by the sending end for the synchronization task. The unstructured files to be synchronized are divided into blocks by dynamic rules to generate multiple initial data blocks. Corresponding metadata information is added to the initial data blocks, which are generated based on multiple initial data blocks and multiple metadata information.

[0032] The module for reassembling and reassembling synchronized files is used to sort and reassemble multiple target metadata blocks to obtain the reassembled synchronized unstructured files of the synchronization task.

[0033] Based on the same inventive concept, the present invention also provides an electronic device, comprising: at least one processor and a memory; wherein the memory and the processor are connected via a bus;

[0034] The memory is used to store one or more programs;

[0035] When the one or more programs are executed by the at least one processor, the method for sending unstructured files between internal and external networks, or the method for receiving unstructured files between internal and external networks, as described above, are implemented.

[0036] Based on the same inventive concept, the present invention also provides a readable storage medium having an executable program stored thereon, wherein when the executable program is executed, it implements the aforementioned method for sending unstructured files on internal and external networks, or the method for receiving unstructured files on internal and external networks.

[0037] Compared with the closest existing technology, the present invention has the following beneficial effects:

[0038] This invention provides a synchronization system for unstructured files between internal and external networks, comprising a sending end and a receiving end communicating via an internal / external network isolation device. The sending end acquires the unstructured file to be synchronized; it dynamically segments the unstructured file using rules to generate multiple initial data blocks; it adds corresponding metadata information to the initial data blocks, and generates multiple target metadata blocks based on the multiple initial data blocks and the metadata information. The receiving end acquires the multiple target metadata blocks using a penetration transmission technology based on the internal / external network isolation device; it sorts and reassembles the multiple target metadata blocks to obtain the reassembled synchronized unstructured file for the synchronization task. This invention generates target metadata blocks by dynamically segmenting the unstructured file to be synchronized, improving transmission efficiency through segmented transmission, and ensuring data security and reducing transmission risks through the internal / external network isolation device using target metadata blocks. Attached Figure Description

[0039] Figure 1 A structural diagram of the synchronization system for unstructured files on internal and external networks provided by this invention;

[0040] Figure 2 A schematic diagram illustrating the structure for synchronizing unstructured files between internal and external networks provided by this invention;

[0041] Figure 3 A schematic diagram illustrating the data flow during the synchronization of unstructured files between internal and external networks, provided by this invention.

[0042] Figure 4 This is a schematic diagram illustrating the communication configuration of the internal and external network isolation device provided by the present invention.

[0043] Figure 5 A monitoring diagram of the application monitoring module provided by this invention;

[0044] Figure 6 A flowchart illustrating the method for sending unstructured files between internal and external networks provided for the invention;

[0045] Figure 7 A structural diagram of the device for sending unstructured files between internal and external networks provided by the present invention;

[0046] Figure 8 A flowchart illustrating the method for receiving unstructured files on internal and external networks provided by this invention;

[0047] Figure 9 A structural diagram of the device for receiving unstructured files from internal and external networks provided by the present invention;

[0048] Figure 10 A schematic diagram of the electronic device provided by the present invention. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0050] Before describing the synchronization system, sending method, and receiving method for unstructured files on internal and external networks provided by this invention, a brief introduction to the technical terms and key terminology involved in this invention will be provided.

[0051] Data block segmentation and metadata appending: The sending end segments the unstructured file into multiple data blocks according to preset rules and appends metadata (such as sequence number) to each data block to ensure that the transmission process is traceable and recombinable.

[0052] SQL (Structured Query Language) tunneling transmission: Data blocks are converted into SQL statements using the driver package of the internal / external network isolation device, and then written into the binary fields of the receiving end's database, bypassing the isolation device. This technology utilizes a database channel to bypass traditional file transfer limitations, enabling cross-network data flow.

[0053] Binary field storage: The receiving database uses binary fields (BLOB type, Binary Large Object, a special data type for storing large binary data) to store the data blocks transmitted through the tunnel, preserving the original file format and metadata, and avoiding the complexity of structured data conversion.

[0054] Sequential Reassembly Mechanism: The receiving end periodically reads newly added data blocks from the binary field through the consumer program, sorts them according to the metadata, and reassembles them to restore the original unstructured file. This process relies on an incremental synchronization strategy and only processes newly added or modified data.

[0055] Internal and external network isolation devices: physical or logical isolation devices (such as firewalls, dedicated storage area networks) control the data exchange path between internal and external networks, allowing only transmission through SQL or specific protocols to ensure network boundary security.

[0056] SAN (Storage Area Network Mapping): Configure SAN storage in an isolated area, divide shared storage volumes and map them to internal and external network servers respectively, and achieve bidirectional data synchronization through access control (such as external network read and write, internal network read only).

[0057] Unformatted data model: Unstructured data is stored using a non-relational model (NoSQL, Not Only SQL), which supports flexible file formats (such as images, logs, videos, documents, etc.), unlike the formatted storage of traditional relational databases.

[0058] Incremental synchronization strategy: Only synchronize changed data blocks since the last transmission. Combined with version control and conflict detection (such as timestamp comparison), it reduces network bandwidth consumption and improves efficiency. Once synchronization is complete, the data can be recycled.

[0059] High-efficiency transmission channels: By using database protocols (such as SQL) to replace traditional file transfer, the limitations of network isolation policies on file sharing are avoided.

[0060] End-to-end consistency: Based on metadata verification and conflict resolution rules, ensure the integrity of data during the process of cutting, transmission, and reassembly.

[0061] Example 1:

[0062] This invention provides a synchronization system for unstructured files on internal and external networks. Specifically, Figure 1 The diagram illustrates the structure of a system for synchronizing unstructured files between internal and external networks, as provided in an embodiment of the present invention. The system includes a sending end and a receiving end that communicate through an internal / external network isolation device.

[0063] The sending end is used to obtain the unstructured file to be synchronized for the synchronization task; the unstructured file to be synchronized is divided into multiple initial data blocks by dynamic rules; corresponding metadata information is added to the initial data blocks, and multiple target metadata blocks are generated based on the multiple initial data blocks and the multiple metadata information.

[0064] The receiving end is used for penetration transmission technology based on internal and external network isolation devices to obtain multiple target metadata blocks; the multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronous unstructured file of the synchronization task.

[0065] The unstructured files to be synchronized in this invention have a large capacity and include one or more of the following: image files, video files, design drawing files, and log files.

[0066] In some optional implementations, the specific process by which the sending end obtains multiple target metadata blocks from the acquired unstructured file to be synchronized through the above processing can be as follows: read the unstructured file to be synchronized, divide it into multiple data blocks according to dynamic rules, attach metadata information including block sequence number, block hash value, file identifier, etc. to each data block, and store the data block with metadata information in binary form in a specified field of the intranet database.

[0067] The target metadata block includes a file information table and a file data table. The file information table stores metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time. The file data table stores the initial data block after binary processing, which includes one or more of the following: file ID, block number, file content, and file path.

[0068] In this invention, the sending end and the receiving end are isolated by an internal and external network isolation device. The unstructured file to be synchronized by the sending end is sent to the receiving end completely and without loss through the internal and external network isolation device to complete the synchronization task.

[0069] Specifically, the sending end can be an intranet environment or an extranet environment; correspondingly, the receiving end can be an extranet environment or an intranet environment. The logic for synchronizing unstructured files from an intranet environment to an extranet environment is basically the same as that for synchronizing them from an extranet environment to an intranet environment. The difference is that this invention only has a database in the intranet environment, i.e., an intranet database.

[0070] When the sending end is an intranet environment and the receiving end is an extranet environment, that is, when synchronizing unstructured files to be synchronized in the intranet environment to the extranet environment, the sending end stores multiple target metadata blocks in the intranet database. Then, the receiving end, based on the intranet-extranet isolation device, periodically accesses the intranet database to obtain multiple target metadata blocks.

[0071] Specifically, when the sending end is located in an intranet environment and the receiving end is located in an extranet environment, the sending end is also used to: store multiple target metadata blocks in the intranet database; the receiving end is specifically used to: periodically access the intranet database to obtain multiple target metadata blocks by polling or monitoring logs based on the penetration transmission technology of the intranet isolation device.

[0072] When the sending end is an external network environment and the receiving end is an internal network environment, that is, when synchronizing unstructured files to be synchronized in the external network environment to the internal network environment, the receiving end stores multiple target metadata blocks in the internal network database based on the internal and external network isolation device, and then periodically accesses the internal network database to obtain multiple target metadata blocks.

[0073] Specifically, when the sending end is located in an external network environment and the receiving end is located in an internal network environment, the receiving end is used to: store multiple target metadata blocks in the internal network database based on the penetration transmission technology of the internal and external network isolation device; and periodically access the internal network database to obtain multiple target metadata blocks by polling or monitoring logs.

[0074] The internal and external network isolation device, also known as the information security network isolation device, includes an isolation device driver package, an SQL proxy service, and an isolation device client. The SQL proxy service, as the core service of the information security isolation device, implements a tunneling transmission technology. This involves using the driver package of the internal and external network isolation device to convert the database into SQL statements, which are then written into the binary fields of the receiving database, bypassing the internal and external network isolation devices. This tunneling transmission technology utilizes a database channel to bypass traditional file transfer restrictions, enabling cross-network data flow.

[0075] To further facilitate administrators in viewing and maintaining each task and keeping track of task progress in a timely manner, an application monitoring module is also set up in the intranet environment of the internal and external network unstructured file synchronization system. This module allows administrators to view various types of data during task execution.

[0076] Specifically, the application monitoring module queries the process data generated during the execution of the synchronization task and monitors the execution progress of the synchronization task based on the process data; the process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was successfully synchronized, and file logs.

[0077] In some optional implementations, to further improve transmission efficiency, the peak and off-peak periods of task execution can be determined based on the number of successfully executed tasks and the time taken by the application monitoring module in the intranet environment within a given period. When service usage reaches its peak, the sending end can dynamically reduce the number of tasks to be executed and the size of the blocks; when service usage is at its low peak and the network condition is good, the sending end can appropriately increase the number of tasks to be executed within a given period.

[0078] Specifically, the application monitoring module determines the peak and valley periods of business usage based on the number of successfully synchronized files within a preset period; correspondingly, the sending end dynamically adjusts the block size of the unstructured files to be synchronized according to the peak and valley periods; based on the adjusted block size, the unstructured files to be synchronized are divided into blocks according to dynamic rules to generate multiple initial data blocks.

[0079] To further illustrate the synchronization system for unstructured files on internal and external networks provided by this invention, a specific example is provided below. Figure 2 This is a schematic diagram illustrating the structure for synchronizing unstructured files between internal and external networks, as provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the data flow during the synchronization of unstructured files between internal and external networks, provided as an embodiment of the present invention. For example... Figure 2 , Figure 3 As shown, the internal and external network unstructured file synchronization system consists of three parts: the internal network environment, the information security network isolation device, and the external network environment.

[0080] The system is divided into three parts: Part 1, the intranet environment, includes: intranet file storage service (OSS, Object Storage Service), intranet file synchronization microservice, and intranet database. Part 2, the information security isolation device service, includes: isolation device driver package, SQL agent service, and isolation device client. Part 3, the extranet environment, includes: extranet file storage service (OSS, Object Storage Service) and extranet file synchronization microservice.

[0081] The internal network environment and the external network environment exchange data through information security isolation devices. For example... Figure 2 As shown, both the intranet file synchronization microservice and the extranet file synchronization microservice are located in a three-tier cloud data center, separated from the intranet office domain and the extranet office domain by the intranet user boundary and the extranet user boundary, respectively. The boundary between the intranet environment and the extranet environment is separated by an information security isolation device.

[0082] like Figure 3 As shown, when it is necessary to synchronize video data from the intranet environment to the external network environment, perform the following steps:

[0083] N1: The intranet file synchronization microservice retrieves video data stored in the intranet file storage service;

[0084] N2: The intranet file microservice segments intranet video data and stores it in the intranet database;

[0085] N3: Through the information security isolation device, based on the block sequence number, the external network file synchronization microservice reads the data block file and sorts and reassembles it;

[0086] N4: After reconstruction, the video will be stored in an external file storage service.

[0087] When you need to synchronize video data from an external network environment to an internal network environment, perform the following steps:

[0088] Y1: The external network file synchronization microservice retrieves video data stored in the external network file storage service;

[0089] Y2: The external network file synchronization microservice segments external network video data into blocks and stores them in an internal network database through an information security isolation device;

[0090] Y3: Based on the block sequence number, the intranet file synchronization microservice reads the data block file and sorts and reassembles it.

[0091] Y4: After reassembly, the video will be stored in the intranet file storage service.

[0092] Since the internal network file synchronization microservice and the external network file synchronization microservice, as well as the internal network file storage service and the external network file storage service, have the same structure, differing only in their internal and external network environments, they will be collectively referred to as file synchronization microservice and file storage service when there is no need to differentiate between them in the future.

[0093] The following section uses the example of synchronizing data from an intranet environment to an extranet environment to illustrate in detail the intranet-extranet unstructured file synchronization system provided by this invention.

[0094] The file storage service in the first part of the intranet environment can use the State Grid Alibaba Cloud Component Object Storage (OSS) in the intranet environment, which mainly stores unstructured files (images, videos, documents). In this invention, unstructured files refer to large video files with a capacity of more than 500M.

[0095] The file synchronization technology of the intranet file synchronization microservice adopts a microservice architecture and is deployed in the intranet Alibaba Cloud EDAS (Enterprise Distributed Application Service). The main functions of the intranet file synchronization microservice include OSS object storage, integration with NoSQL driver packages, and file management. File management mainly involves file reading and writing, file storage, file sharding, file verification, and file reassembly. When transferring unstructured files from the intranet environment to the external network environment, the intranet file synchronization microservice reads the unstructured files stored in the intranet file storage service and performs sharding using dynamic rules to generate multiple data blocks. Metadata information, including block sequence number, block hash value, and file identifier, is appended to each data block. The data blocks with metadata are stored in binary form in a specified field of the intranet database; the specified field type can be LONGBLOB. The sharding can be configured visually, and the size of a data block can be 50MB. The code for sharding is as follows:

[0096]

[0097] The following are annotations:

[0098] Line 1: Creates an array whose size is a variable that can be dynamically adjusted, for example, to 10M;

[0099] Lines 2 and 3: Define two local variables;

[0100] Line 5: Creates a List collection to store object data;

[0101] Line 6: Loop statement, looping through the sample video file in 10MB chunks;

[0102] Lines 7-11: The chunked file is set as a parameter to the File property of the hrCmptInoutSyncFile object. Each time the loop is repeated, the chunk number is incremented by 1, and then the above information is stored in the database. The storage is repeated in sequence. If the file is not read, -1 is returned, the loop is skipped, and the next task is read.

[0103] The database server is deployed in an intranet environment, i.e., an intranet database. The intranet database contains two entity tables, which are temporary storage tables for data blocks generated by the intranet file synchronization microservice. The two tables are primarily a file information table and a file data table. The file information table records file metadata, such as file ID, file name, file type, file size, number of blocks, and insertion storage time. The file data table stores binary data block files, with core fields including file ID, block number, and file content. Table 1 shows an example of the file information table, and Table 2 shows an example of the file data table. The specific table structure information is as follows:

[0104]

[0105]

[0106] Table 1

[0107]

[0108]

[0109] Table 2

[0110] The second part, the information security isolation device service, includes: isolation device driver package, SQL agent service, and isolation device client.

[0111] Among them, the isolation device driver package is a customized dependency package of the isolation device. This dependency package has poor compatibility with the LONGBLOB data type. The external network file synchronization microservice connects to the information security isolation device service through this driver package.

[0112] As a core service in the information security isolation device service, the SQL proxy service mainly acts as a proxy to connect to the internal network environment database service and return the queried data blocks to the external network file synchronization microservice.

[0113] The isolation device client primarily provides users with a visual interface to facilitate better configuration of the isolation device. For example... Figure 4 This is a schematic diagram illustrating the configuration of communication between internal and external network isolation devices. The main steps are as follows:

[0114] (1) Open the isolation device client;

[0115] (2) Configure the SQL Agent Server;

[0116] (3) After successfully logging in, you will enter the interface;

[0117] (4) Configure the actual database information;

[0118] (5) Configure virtual database information;

[0119] (6) Configure SQL Agent application information;

[0120] (7) Configure the SQL Agent application server;

[0121] (8) Data source information for file synchronization service.

[0122] The second part, the external network environment section, includes: external network file synchronization microservice and external network file storage service (OSS).

[0123] The external network file synchronization microservice shares the same structure as the internal network file synchronization microservice, primarily used for integrating OSS object storage, integrating isolation device driver packages, and file management. File management mainly involves file reading and writing, file storage, file sharding, file verification, and file reassembly. When synchronizing data from the internal network environment to the external network environment, the external network file synchronization microservice periodically accesses the internal network database through polling or monitoring logs to obtain the binary content and metadata information of newly added databases. It then sorts and reassembles the database based on the block sequence numbers in the metadata. Furthermore, it can verify the integrity of the metadata using block hash values, ultimately restoring the original files and storing them in the external network file storage service. The code for the reassembly technology is as follows:

[0124] List streams=new ArrayList();

[0125] For(hrCmptInoutSyncFile hrCmptInoutSyncFile:hrCmptInoutSyncFiles){

[0126] hrCmptInoutSyncFilefileInfo=hrCmptInoutSyncFileMapper.selectById(hrCmptInoutSyncFile.getRecordId());

[0127] ByteArrayInputStream byteArrayInputStream=new ByteArrayInputStream(fileInfo.getFile());

[0128] Streams.add(byteArrayInputStream);

[0129] fileName=fileInfo.getFileName();

[0130] }

[0131] InputStream fileByte=new SequenceInputStream(collections.enumeration(streams));

[0132] The following are annotations:

[0133] Line 1: Creates a list collection to store the chunked files;

[0134] Line 2: This is a loop statement that queries the database for a list of partitioned data and iterates through the data in sequence.

[0135] Lines 3-7: Query detailed information about the chunked data, then convert the chunked files into binary file stream objects and store them into a list collection in sequence;

[0136] Line 8: Merges and reassembles the fragmented files in the list collection into a complete stream file and uploads it to OSS object storage.

[0137] In some alternative implementations, data consistency can be ensured by using the overall checksum of the received file, such as an MD5 code (Message Digest Algorithm 5), and the value recorded by the sender.

[0138] External network file storage services can use the State Grid Alibaba Cloud Component Object Storage (OSS) in the external network environment, mainly storing unstructured files (images, videos, documents). In this invention, large video files transmitted from the internal network can be stored.

[0139] In some optional implementations, fault tolerance mechanisms are also provided. Redundancy is incorporated into the internal network storage and external network restoration of processed data blocks. When an exception occurs during program execution, the task ID can be marked, and the program will retry after 5 minutes. When the number of retries reaches a preset number (e.g., 3), the task is marked, removed from the pending tasks list, and a system message is sent. The task's details can be viewed in the internal network application monitoring function, including execution time, task ID, file name, synchronization success status, error logs, etc. The visualization page is available in the application monitoring function. All processed tasks are added to the audit log, recording the details of each executed task. When the data volume is large, database sharding and table partitioning can be used in data processing and query design.

[0140] In some optional implementations, application monitoring functionality is also provided. At the application monitoring level, a visual, real-time interface can be provided, allowing administrators to easily view and maintain each task and promptly grasp its progress. Dynamic task adjustments are also possible, with the segmentation strategy dynamically adjusted based on the number of successfully executed tasks and their execution time within a cycle. In actual application monitoring, transmission efficiency was improved by 38%. Figure 5 This is a monitoring diagram of the application monitoring module provided by the present invention. Figure 5 The diagram illustrates the monitoring of data transmission from the intranet environment to the extranet environment, including the source of each data block, upload start time, upload end time, upload log, number of uploads, upload status, download start time, download end time, and download status.

[0141] In some optional implementations, depending on the characteristics of the business scenario, when the business usage reaches its peak, the number of tasks to be executed and the size of the blocks can be dynamically reduced; during the off-peak period, when the network condition is good, the number of tasks to be executed can be increased.

[0142] In some optional implementations, after uploading a video file in the intranet environment, it needs to be played frame by frame to review the video content and mark it if it passes the review. Based on the applicable scope of the video, it is determined whether the video should be synchronized to the external network environment. During synchronization, the data is transmitted using the data block principle. After the external network reassembles the data according to the block sequence number, the integrity of the video can be verified by MD5 code to prevent tampering during transmission.

[0143] In this invention, unstructured files are divided into multiple data blocks according to a preset fixed or dynamic size. Each data block is appended with metadata information including a block number, a block hash value, and a file identifier. The metadata data blocks are stored in binary form in a LONGBLOB field. The data blocks are sorted and reorganized according to their block numbers, and their integrity is verified by MD5. The invention also includes multiple functions such as fault tolerance mechanisms, retry counts, application monitoring and control, and security assurance.

[0144] Compared with existing technologies, this invention uses block-splitting technology to break down large unstructured files (such as video footage) into independent data blocks, enabling parallel transmission and intelligent scheduling. Based on the block-splitting technology, a block verification mechanism generates a block sequence value for each data block and stores it in NoSQL. This invention is suitable for enterprise-level file data block penetration through isolation devices in internal and external network isolation environments, and is designed with fault tolerance mechanisms and application monitoring functions.

[0145] During the design verification of this invention, multiple sets of data samples were used, and some sample verification results are as follows:

[0146] The file information is shown in Table 3:

[0147]

[0148]

[0149] The file data blocks in Table 3 are shown in Table 4:

[0150]

[0151] Table 4

[0152] Based on the principle of segmenting unstructured files between internal and external networks, it can be found through experiments that the synchronization system for unstructured files between internal and external networks provided by this invention is basically consistent with the actual situation.

[0153] Example 2:

[0154] Based on the same inventive concept, this invention also provides a method for sending unstructured files between internal and external networks, such as... Figure 6 The diagram shown is a flowchart illustrating the method for sending unstructured files between internal and external networks provided by this invention. The method includes:

[0155] S601: Obtain the unstructured file to be synchronized for the synchronization task;

[0156] S602: For synchronized unstructured files, multiple initial data blocks are generated by dynamically dividing and splitting them according to rules;

[0157] S603: Add corresponding metadata information to the initial data block, generate multiple target metadata blocks based on multiple initial data blocks and multiple metadata information, so that the receiving end can obtain multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; sort and reorganize the multiple target metadata blocks to obtain the reorganized synchronous unstructured file of the synchronization task.

[0158] Preferably, when the sending end of the method for sending unstructured files between internal and external networks is located in an internal network environment and the receiving end is located in an external network environment, the method further includes: storing multiple target metadata blocks in an internal network database.

[0159] Preferably, when the sending end of the method for sending unstructured files between internal and external networks is located in an internal network environment, the method further includes:

[0160] The application monitoring module in the intranet environment queries the process data generated during the execution of the synchronization task, and monitors the execution progress of the synchronization task based on the process data. The process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was successfully synchronized, and file logs.

[0161] Preferably, the method further includes:

[0162] The peak and off-peak periods of business usage are determined based on the number of files successfully synchronized within a preset period.

[0163] The block size of the unstructured file to be synchronized is dynamically adjusted according to the peak and valley periods; based on the adjusted block size, the unstructured file to be synchronized is divided into blocks according to dynamic rules to generate multiple initial data blocks.

[0164] Preferably, the unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files;

[0165] The target metadata block includes a file information table and a file data table; the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time.

[0166] The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

[0167] Example 3:

[0168] Based on the same inventive concept, the present invention also provides a device for sending unstructured files between internal and external networks, such as... Figure 7 The diagram shown is a structural diagram of the device for sending unstructured files between internal and external networks provided by the present invention. The sending device 700 includes:

[0169] The file acquisition module 701 is used to acquire the unstructured files to be synchronized for the synchronization task;

[0170] The block splitting processing module 702 is used to split unstructured files to be synchronized into multiple initial data blocks by using dynamic rules.

[0171] The target metadata block generation module 703 is used to add corresponding metadata information to the initial data block, generate multiple target metadata blocks based on multiple initial data blocks and multiple metadata information, so that the receiving end can obtain multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; sort and reorganize the multiple target metadata blocks to obtain the reorganized synchronous unstructured file of the synchronization task.

[0172] Preferably, when the sending device for unstructured files between the internal and external networks is located in the internal network environment and the receiving end is located in the external network environment, the device is also used to: store multiple target metadata blocks in the internal network database.

[0173] Preferably, when the device for sending unstructured files between internal and external networks is located in an internal network environment, the device further includes an application monitoring module for:

[0174] Query the process data generated during the execution of the synchronization task, and monitor the execution progress of the synchronization task based on the process data; the process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was synchronized successfully or not, and file logs.

[0175] Preferably, the application monitoring module is also used for:

[0176] The peak and off-peak periods of business usage are determined based on the number of files successfully synchronized within a preset period.

[0177] The block size of the unstructured file to be synchronized is dynamically adjusted according to the peak and valley periods; based on the adjusted block size, the unstructured file to be synchronized is divided into blocks according to dynamic rules to generate multiple initial data blocks.

[0178] Preferably, the unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files;

[0179] The target metadata block includes a file information table and a file data table; the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time.

[0180] The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

[0181] Example 4:

[0182] Based on the same inventive concept, this invention also provides a method for receiving unstructured files on internal and external networks, such as... Figure 8 The diagram shown is a flowchart illustrating the method for receiving unstructured files on internal and external networks provided by this invention. The method includes:

[0183] S801: Based on the penetration transmission technology of the internal and external network isolation device, it obtains multiple target metadata blocks. The multiple target metadata blocks are the unstructured files to be synchronized by the sending end for the synchronization task. The unstructured files to be synchronized are divided into blocks by dynamic rules to generate multiple initial data blocks. Corresponding metadata information is added to the initial data blocks. The data is generated based on multiple initial data blocks and multiple metadata information.

[0184] S802: Sort and reorganize multiple target metadata blocks to obtain the reorganized synchronized unstructured file of the synchronization task.

[0185] Preferably, when the receiving end of the method for receiving unstructured files on internal and external networks is located in an external network environment and the sending end is located in an internal network environment, the method further includes:

[0186] Based on the penetration transmission technology of the internal and external network isolation device, the internal network database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

[0187] Preferably, when the receiving end of the method for receiving unstructured files on internal and external networks is located in an internal network environment and the sending end is located in an external network environment, the method further includes:

[0188] Based on the penetration transmission technology of the internal and external network isolation device, multiple target metadata blocks are stored in the internal network database;

[0189] The internal database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

[0190] Preferably, when the receiving end used in the method for receiving unstructured files between internal and external networks is located in an internal network environment, the method further includes:

[0191] The application monitoring module in the intranet environment queries the process data generated during the execution of the synchronization task, and monitors the execution progress of the synchronization task based on the process data. The process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was successfully synchronized, and file logs.

[0192] Preferably, the method further includes:

[0193] The peak and off-peak periods of business usage are determined based on the number of files successfully synchronized within a preset period.

[0194] The block size of the unstructured file to be synchronized is dynamically adjusted according to the peak and valley periods; based on the adjusted block size, the unstructured file to be synchronized is divided into blocks according to dynamic rules to generate multiple initial data blocks.

[0195] Preferably, the unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files;

[0196] The target metadata block includes a file information table and a file data table; the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time.

[0197] The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

[0198] Example 5:

[0199] Based on the same inventive concept, the present invention also provides a device for receiving unstructured files on internal and external networks, such as... Figure 9The diagram shown is a structural diagram of the receiving device for unstructured files on internal and external networks provided by the present invention. The sending device 900 includes:

[0200] The target metadata block acquisition module 901 is used to acquire multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device. The multiple target metadata blocks are the unstructured files to be synchronized by the sending end for the synchronization task. The unstructured files to be synchronized are divided into blocks by dynamic rules to generate multiple initial data blocks. Corresponding metadata information is added to the initial data blocks, which are generated based on multiple initial data blocks and multiple metadata information.

[0201] The reassembled synchronization file acquisition module 902 is used to sort and reassemble multiple target metadata blocks to obtain the reassembled synchronization unstructured file of the synchronization task.

[0202] Preferably, when the receiving device for unstructured files on the internal and external networks is located in the external network environment and the sending end is located in the internal network environment, the device is also used for:

[0203] Based on the penetration transmission technology of the internal and external network isolation device, the internal network database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

[0204] Preferably, when the receiving device for unstructured files on the internal and external networks is located in the internal network environment and the sending end is located in the external network environment, the device is also used for:

[0205] Based on the penetration transmission technology of the internal and external network isolation device, multiple target metadata blocks are stored in the internal network database;

[0206] The internal database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

[0207] Preferably, when the device for receiving unstructured files from internal and external networks is located in an internal network environment, the device further includes an application monitoring module for:

[0208] Query the process data generated during the execution of the synchronization task, and monitor the execution progress of the synchronization task based on the process data; the process data includes one or more of the following: the execution time of the synchronization task, file ID, file name, whether the file was synchronized successfully or not, and file logs.

[0209] Preferably, the application monitoring module is also used for:

[0210] The peak and off-peak periods of business usage are determined based on the number of files successfully synchronized within a preset period.

[0211] The block size of the unstructured file to be synchronized is dynamically adjusted according to the peak and valley periods; based on the adjusted block size, the unstructured file to be synchronized is divided into blocks according to dynamic rules to generate multiple initial data blocks.

[0212] Preferably, the unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files;

[0213] The target metadata block includes a file information table and a file data table; the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time.

[0214] The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

[0215] Example 6:

[0216] Based on the same inventive concept, such as Figure 10 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0217] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in a readable storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of the method for sending unstructured files between internal and external networks or the method for receiving unstructured files between internal and external networks in the above embodiments.

[0218] Example 7:

[0219] Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the readable storage medium here can include both the built-in storage medium of the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the storage medium to implement the steps of the above-described methods for sending or receiving unstructured files between internal and external networks.

[0220] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0222] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0223] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A synchronization system for unstructured files on internal and external networks, characterized in that, This includes the sending and receiving ends that communicate through internal and external network isolation devices; The sending end is used to obtain the unstructured file to be synchronized for the synchronization task; The unstructured file to be synchronized is divided into multiple initial data blocks using dynamic rule-based block segmentation. Add corresponding metadata information to the initial data block, and generate multiple target metadata blocks based on the multiple initial data blocks and the multiple metadata information; The receiving end is used to acquire multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; Multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronized unstructured file for the synchronization task.

2. The system according to claim 1, characterized in that, When the sending end is located in an intranet environment and the receiving end is located in an extranet environment, the sending end is also used to: store the plurality of target metadata blocks in an intranet database; The receiving end is specifically used to: periodically access the internal network database and obtain the multiple target metadata blocks by means of polling or monitoring logs, based on the penetration transmission technology of the internal and external network isolation device.

3. The system according to claim 1, characterized in that, When the sending end is located in an external network environment and the receiving end is located in an internal network environment, the receiving end is specifically used for: Based on the penetration transmission technology of the aforementioned internal and external network isolation device, the multiple target metadata blocks are stored in the internal network database; The internal network database is accessed periodically by polling or monitoring logs to obtain multiple target metadata blocks.

4. The system according to claim 1, characterized in that, The system also includes an application monitoring module located in the intranet environment; The application monitoring module is specifically used to: query the process data generated during the execution of the synchronization task, and monitor the execution progress of the synchronization task based on the process data; the process data includes one or more of the following: synchronization task execution time point, file ID, file name, whether the file was successfully synchronized, and file log.

5. The system according to claim 4, characterized in that, The application monitoring module is also used to: determine the peak and valley periods of business usage based on the number of successfully synchronized files within a preset period; The sending end is also used to: dynamically adjust the block size of the unstructured file to be synchronized according to the peak and valley periods; Based on the adjusted block size, the unstructured file to be synchronized is divided into blocks using dynamic rules to generate multiple initial data blocks.

6. The system according to any one of claims 1-5, characterized in that, The unstructured files to be synchronized include one or more of the following: image files, video files, design drawing files, and log files. The target metadata block includes a file information table and a file data table; wherein, the file information table is used to store metadata information, which includes one or more of the following: file ID, file name, file type, file size, number of blocks, and insertion storage time; The file data table is used to store the initial data blocks after binary encoding. The initial data blocks include one or more of the following: file ID, block number, file content, and file path.

7. A method for sending unstructured files between internal and external networks, characterized in that, include: Retrieve the unstructured files to be synchronized for the synchronization task; The unstructured file to be synchronized is divided into multiple initial data blocks using dynamic rule-based block segmentation. Add corresponding metadata information to the initial data block, and generate multiple target metadata blocks based on multiple initial data blocks and multiple metadata information, so that the receiving end can obtain multiple target metadata blocks based on the penetration transmission technology of the internal and external network isolation device; Multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronized unstructured file for the synchronization task.

8. A method for receiving unstructured files on internal and external networks, characterized in that, include: Based on the penetration transmission technology of the aforementioned internal and external network isolation device, multiple target metadata blocks are obtained. These multiple target metadata blocks are unstructured files to be synchronized by the sending end for obtaining synchronization tasks. The unstructured file to be synchronized is divided into multiple initial data blocks using dynamic rule-based block segmentation. Add corresponding metadata information to the initial data block, which is generated based on multiple initial data blocks and multiple metadata information; Multiple target metadata blocks are sorted and reorganized to obtain the reorganized synchronized unstructured file for the synchronization task.

9. An electronic device, characterized in that, include: At least one processor and memory; The memory and processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the method for sending unstructured files on internal and external networks as described in claim 7, or the method for receiving unstructured files on internal and external networks as described in claim 8, is implemented.

10. A readable storage medium, characterized in that, It contains an executable program, which, when executed, implements the method for sending unstructured files between internal and external networks as described in claim 7, or the method for receiving unstructured files between internal and external networks as described in claim 8.

Images