Cross-network structured and unstructured data synchronization method, device and equipment
By building an FTP server and network gate, combined with JSON and SHELL scripts, the synchronization of cross-network structured and unstructured data is achieved, solving the problem of data interaction difficulties in network isolation environment, and improving the security and efficiency of data propagation.
Patent Information
- Application Number
- CN202411036016.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-26
AI Technical Summary
In a network isolation environment, it is difficult for enterprises to achieve efficient synchronization of structured and unstructured data across networks, resulting in difficulty in data interaction.
By building an Internet FTP server, an intranet FTP server and a network gate, and configuring network gate file synchronization rules, using JSON and SHELL script timing tasks to synchronize structured and unstructured data, realizing cross-network data ferrying.
In the absence of necessary network environments and tools, the synchronization of structured and unstructured data is achieved, solving the problem of data synchronization and enhancing the security and efficiency of data propagation.
Smart Images

Figure CN119030994A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data synchronization, and in particular to a method, device and equipment for synchronizing structured and unstructured data across networks. Background Art
[0002] In order to protect the security of core data assets, enterprises currently choose network isolation when building networks, that is, to physically separate the external internet network from the internal and external business networks, in order to resist viruses and hacker intrusions, prevent the spread and leakage of important data, and enhance the security of asset data. As the degree of enterprise informatization increases, the importance of core data protection is also increasing. However, after network isolation, data interaction between the internal network and the external network or between two different networks has become a problem. The common method is to introduce a network gate, configure access transmission rules in the network gate, and conduct cross-network communication. However, there are still problems such as difficulty in business data flow during cross-network development. Summary of the invention
[0003] The main purpose of the present invention is to provide a method, device and equipment for synchronizing structured and unstructured data across networks, aiming to solve at least one of the above-mentioned technical problems.
[0004] To achieve the above object, the present invention provides a method for synchronizing structured and unstructured data across networks, comprising:
[0005] Set up an Internet FTP server, an intranet FTP server, and a network gatekeeper, and configure the network gatekeeper file synchronization rules;
[0006] Upon receiving the data synchronization instruction, the structured data and the unstructured data in the Internet domain are saved to the Internet FTP server respectively;
[0007] Transferring files from the Internet FTP server to the Intranet FTP server based on the network gate file synchronization rules;
[0008] Scan the structured data directory of the intranet FTP server based on the JSON scheduled parsing task and perform structured data synchronization;
[0009] Based on the SHELL script scheduled task, the unstructured data directory of the intranet FTP server is searched and the unstructured data synchronization is performed.
[0010] In some embodiments, the steps of respectively building an Internet FTP server, an intranet FTP server, and a network gatekeeper, and configuring network gatekeeper file synchronization rules include:
[0011] Set up an Internet FTP server in the Internet domain;
[0012] An intranet FTP server is built in the intranet domain; wherein the Internet domain and the intranet domain are network isolated;
[0013] A network gate is built and network gate file synchronization rules are configured so that the Internet FTP server and the intranet FTP server can perform cross-network file transfer regularly based on the network gate file synchronization rules.
[0014] In some embodiments, the method further comprises:
[0015] A JAVA program is deployed in an unstructured data storage server in an Internet domain; wherein the JAVA program is used to provide an API for executing a SYNC command so as to synchronize files in the unstructured data storage server to an unstructured data directory in an Internet FTP server.
[0016] In some embodiments, upon receiving the data synchronization instruction, saving the structured data and the unstructured data in the Internet domain to the Internet FTP server respectively includes:
[0017] When the unstructured data synchronization demand is triggered, call the API to assemble the SYNC command;
[0018] executing the SYNC command to copy the unstructured data file from the unstructured data storage server;
[0019] Uploading the unstructured data file and the directory structure to the unstructured data directory of the Internet FTP server;
[0020] When the structured data synchronization demand is triggered, a preset rule JSON file is generated and the preset rule JSON file is uploaded to the Internet FTP server.
[0021] In some embodiments, generating a preset rule JSON file and uploading the preset rule JSON file to the Internet FTP server includes:
[0022] saving the structured data to a database in the Internet domain;
[0023] Convert the structured data of the database into a preset rule JSON file based on a preset code; wherein the preset rule JSON file includes a target table name, a synchronization field, and a field value;
[0024] Upload the preset rule JSON file to the structured data directory of the Internet FTP server.
[0025] In some embodiments, the ferrying of the file on the Internet FTP server to the intranet FTP server based on the gatekeeper file synchronization rule includes:
[0026] The ferrying program based on the network gate executes a ferrying task once at a first preset time interval; wherein the ferrying task is to ferry the file of the Internet FTP server to the intranet FTP server based on the network gate file synchronization rule.
[0027] In some embodiments, the JSON-based scheduled parsing task scans the structured data directory of the intranet FTP server and performs structured data synchronization, including:
[0028] Scanning the structured data directory of the intranet FTP server at a second preset time interval based on the JSON timing parsing task;
[0029] When a target JSON file is scanned in the structured data directory, the target JSON file is parsed and SQL is assembled to obtain parsed data and save the parsed data to a file server in the intranet domain;
[0030] After parsing is complete, the target JSON file of the intranet FTP server is deleted to complete the synchronization of structured data.
[0031] In some embodiments, the SHELL script-based scheduled task searches for the unstructured data directory of the intranet FTP server and performs unstructured data synchronization, including:
[0032] Scanning the unstructured data directory of the intranet FTP server at a third preset time interval based on a SHELL script scheduled task;
[0033] When a target unstructured data file is scanned in the unstructured data directory, an RSYNC command is executed and the target unstructured data file is cut to a file server in the intranet domain to complete the synchronization of the unstructured data.
[0034] In addition, to achieve the above-mentioned purpose, the present invention also proposes a cross-network structured and unstructured data synchronization device, comprising:
[0035] FTP server building module, used to build Internet FTP server, intranet FTP server and network gate, and configure network gate file synchronization rules;
[0036] A synchronization trigger module, for saving the structured data and the unstructured data in the Internet domain to the Internet FTP server respectively upon receiving a data synchronization instruction;
[0037] A cross-network ferry module, used for ferrying files from the Internet FTP server to the intranet FTP server based on the network gate file synchronization rule;
[0038] A structured data synchronization module, used to scan the structured data directory of the intranet FTP server and perform structured data synchronization based on the JSON scheduled parsing task;
[0039] The unstructured data synchronization module is used to search the unstructured data directory of the intranet FTP server based on the SHELL script scheduled task and perform unstructured data synchronization.
[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, which includes: a memory, a processor, and a cross-network structured and unstructured data synchronization program stored on the memory and executable on the processor, wherein the cross-network structured and unstructured data synchronization program is configured to implement the cross-network structured and unstructured data synchronization method as described above.
[0041] The present invention provides a cross-network structured and unstructured data synchronization method, the method comprising: respectively building an Internet FTP server, an intranet FTP server and a network gate, and configuring the network gate file synchronization rules; upon receiving a data synchronization instruction, respectively saving the structured data and unstructured data of the Internet domain to the Internet FTP server; ferrying the files of the Internet FTP server to the intranet FTP server based on the network gate file synchronization rules; scanning the structured data directory of the intranet FTP server based on a JSON scheduled parsing task and performing structured data synchronization; searching the unstructured data directory of the intranet FTP server based on a SHELL script scheduled task and performing unstructured data synchronization. In the present invention, by adopting a network gate, an FTP server, a SHELL script and the like, structured and unstructured data are synchronized in different network environments, and the data synchronization problem is solved in the absence of necessary network environments and tools. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;
[0043] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for synchronizing structured and unstructured data across networks according to the present invention;
[0044] Figure 3 A schematic diagram of the network environment setting and cross-network synchronization process involved in the embodiment of the present invention;
[0045] Figure 4This is an example diagram of a SYNC command involved in an embodiment of the present invention;
[0046] Figure 5 This is an example diagram of unstructured data synchronization commands involved in an embodiment of the present invention;
[0047] Figure 6 This is an example diagram of a structured data synchronization command involved in an embodiment of the present invention;
[0048] Figure 7 It is a structural block diagram of an embodiment of a cross-network structured and unstructured data synchronization device of the present invention.
[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0052] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0053] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present invention.
[0054] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0055] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0056] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a cross-network structured and unstructured data synchronization program.
[0057] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the cross-network structured and unstructured data synchronization program stored in the memory 1005 through the processor 1001, and executes the cross-network structured and unstructured data synchronization method provided by an embodiment of the present invention.
[0058] The present invention provides a method, device and equipment for synchronizing structured and unstructured data across networks.
[0059] The embodiment of the present invention provides a method for synchronizing structured and unstructured data across networks. Figure 2 , Figure 2 The present invention is a flowchart of an embodiment of a method for synchronizing structured and unstructured data across networks.
[0060] like Figure 2 As shown, the cross-network structured and unstructured data synchronization method includes:
[0061] Step S100: respectively building an Internet FTP server, an intranet FTP server and a network gate, and configuring the network gate file synchronization rules;
[0062] Step S200: upon receiving a data synchronization instruction, the structured data and the unstructured data in the Internet domain are saved to the Internet FTP server respectively;
[0063] Step S300: transferring the file of the Internet FTP server to the Intranet FTP server based on the network gate file synchronization rule;
[0064] Step S400: Scan the structured data directory of the intranet FTP server based on the JSON scheduled parsing task and perform structured data synchronization;
[0065] Step S500: searching the unstructured data directory of the intranet FTP server based on the SHELL script scheduled task and performing unstructured data synchronization.
[0066] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.
[0067] It is understandable that this embodiment is described by taking the cross-network structured and unstructured data synchronization process between an external network domain (such as the Internet domain) and an internal network domain (such as an enterprise internal business network) as an example.
[0068] In one embodiment, an Internet FTP server, an intranet FTP server and a network gate are respectively established, and network gate file synchronization rules are configured, including: establishing an Internet FTP server in the Internet domain; establishing an intranet FTP server in the intranet domain; wherein the Internet domain and the intranet domain are network isolated; establishing a network gate and configuring network gate file synchronization rules, so that the Internet FTP server and the intranet FTP server regularly perform cross-network file ferrying based on the network gate file synchronization rules.
[0069] It is understandable that "FTP (File Transfer Protocol)" is one of the TCP / IP protocol suite protocols, and its main function is to achieve file transfer between remote hosts with the help of the network.
[0070] Specifically, step 1, set up FTP servers in different network environments: refer to Figure 3The network environment configuration shown in the figure, in the embodiment of the present application, the internal and external network environment configuration is performed, and FTP servers are respectively built in different network environments, and an Internet FTP server is built in the Internet domain (such as Figure 3 Build an intranet FTP server in the intranet domain (such as Figure 3 Intranet FTP server as shown).
[0071] Specifically, step 2 configures the network gate file synchronization rules so that it can perform cross-network file transfer regularly: Figure 3 As shown, a network gate is set up between the Internet and the intranet, a ferry channel is established through the network gate, and cross-network file ferrying is achieved through the ferry channel and the network gate file synchronization rules.
[0072] In one embodiment, the method further includes: deploying a JAVA program in an unstructured data storage server in an Internet domain; wherein the JAVA program is used to provide an API for executing a SYNC command so that the files of the unstructured data storage server are synchronized to the unstructured data directory of an Internet FTP server.
[0073] It should be noted that "SYNC" is a data mirroring backup tool under the LINUX system. It is usually installed by the LINUX system. It can synchronize files on the local host and the remote host (including two synchronization methods: local push to remote and remote pull to local). It can also synchronize files in different local paths, but it cannot synchronize files from remote path 1 to remote path 2. It is necessary to use JAVA code to assemble the SYNC command on the source file side or the target side and execute the command locally.
[0074] Specifically, in step 3, a JAVA program is deployed on the unstructured data storage server, so that the JAVA program provides an API for executing a SYNC command, so that the current server file can be synchronized to a remote FTP server directory.
[0075] For example, Figure 3 As shown, the unstructured data storage server in the Internet domain is deployed with a JAVA program, which copies the unstructured data files of the unstructured data storage server to the unstructured data directory of the Internet FTP server by executing the SYNC command (or RSYNC copy command).
[0076] In one embodiment, when a data synchronization instruction is received, the structured data and unstructured data of the Internet domain are saved to the Internet FTP server respectively, including: when the unstructured data synchronization demand is triggered, calling the API to assemble a SYNC command; executing the SYNC command to copy the unstructured data file from the unstructured data storage server; uploading the unstructured data file and the directory structure to the unstructured data directory of the Internet FTP server; when the structured data synchronization demand is triggered, generating a preset rule JSON file and uploading the preset rule JSON file to the Internet FTP server.
[0077] In one embodiment, generating a preset rule JSON file and uploading the preset rule JSON file to the Internet FTP server includes: saving structured data to a database of the Internet domain; converting the structured data of the database into a preset rule JSON file based on a preset code; wherein the preset rule JSON file includes a target table name, synchronization fields, and field values; and uploading the preset rule JSON file to a structured data directory of the Internet FTP server.
[0078] Specifically, in step 4, a program is written in each network environment. When the unstructured data synchronization demand is triggered, the API is called to assemble a SYNC command and execute the SYNC command to copy the file from the object storage server (i.e., the unstructured data storage server) to the Internet FTP server. For example, the specific command can be referred to Figure 4 .
[0079] Specifically, in step 5, a program is written in each network environment. When the structured data synchronization demand is triggered, a specific rule JSON file (preset rule JSON file) is generated and uploaded to the Internet FTP server. The specific rule refers to the identifiable rules that specify the target table name, synchronization field, field value, etc.
[0080] In one embodiment, the files of the Internet FTP server are ferryed to the intranet FTP server based on the network gate file synchronization rules, including: a ferrying program based on the network gate executes a ferrying task once every first preset time interval; wherein the ferrying task is to ferry the files of the Internet FTP server to the intranet FTP server based on the network gate file synchronization rules.
[0081] Specifically, the ferry program of the network gate performs a ferry task every 5 seconds to ferry the file on the Internet FTP server to the intranet FTP server. It is understandable that the first preset time includes but is not limited to 5 seconds, and this embodiment does not limit this.
[0082] In one embodiment, the structured data directory of the intranet FTP server is scanned based on the JSON timed parsing task and structured data synchronization is performed, including: based on the JSON timed parsing task, the structured data directory of the intranet FTP server is scanned at intervals of a second preset time; when a target JSON file is scanned under the structured data directory, the target JSON file is parsed and SQL is assembled to obtain the parsed data and save the parsed data to a file server in the intranet domain; after the parsing is completed, the target JSON file of the intranet FTP server is deleted to complete the structured data synchronization.
[0083] In one embodiment, the unstructured data directory of the intranet FTP server is searched based on a SHELL script scheduled task and unstructured data synchronization is performed, including: scanning the unstructured data directory of the intranet FTP server at intervals of a third preset time based on the SHELL script scheduled task; when a target unstructured data file is scanned under the unstructured data directory, executing an RSYNC command and cutting the target unstructured data file to a file server in the intranet domain to complete unstructured data synchronization.
[0084] It should be noted that the sequence of steps for unstructured data synchronization and structured data synchronization in this embodiment does not limit the actual operation sequence. The unstructured data synchronization step may be performed first and then the structured data synchronization step, or the structured data synchronization step may be performed first and then the unstructured data synchronization step, or the unstructured data synchronization step and the structured data synchronization step may be performed simultaneously, and this embodiment does not limit this. It is understandable that the second preset time includes but is not limited to 10 seconds, and the third preset time includes but is not limited to 10 seconds, and this embodiment does not limit this.
[0085] In one example, in step 6, each network environment writes a SHELL script to periodically execute the SYNC command to detect and cut the unstructured data files in the target directory of the FTP server (intranet FTP server) to the object storage server, and the unstructured data synchronization is completed. For example, the specific command can be referred to Figure 5 .
[0086] Step 7, each network environment writes a program to parse the JSON file in the target directory of the FTP server (intranet FTP server), assemble the SQL for adding, updating, and deleting, and execute it on the target database (for example, execute the SQL statement directly in the intranet database and insert it into the intranet database table for storage). After the parsing is completed, delete the target JSON file of the intranet FTP server. At this point, the structured data synchronization is completed. For example, the specific commands can be referred to Figure 6 .
[0087] In this embodiment, by adopting network gates, SYNC, FTP, SHELL scripts and codes, synchronization of structured data and unstructured data is achieved in different network environments, providing a way and method for solving data synchronization problems in the absence of necessary network environment and tools.
[0088] It should be noted that the following is a specific example of the cross-network structured and unstructured data synchronization process between the Internet domain and the company's business intranet:
[0089] Suppose that you want to synchronize the names, ages, and JPEG photos of people from the Internet domain to the company's business intranet, where names and ages are structured data and JPEG photos are unstructured data. Suppose that the basic environment such as the network gate and FTP (Internet FTP server, intranet FTP server) has been set up. When the staff clicks the button to trigger data synchronization, the structured data is saved to the database of the current network domain (Internet domain), and then converted into a JSON file by the code and uploaded to the corresponding structured data file directory of the Internet FTP server; the unstructured data will be uploaded to the file server of the current network domain (Internet domain), and then the API will be called to assemble the SYNC command and execute it, and the unstructured data files on the file server will be uploaded to the corresponding unstructured file directory of the Internet FTP server together with the directory structure.
[0090] The ferry program of the network gate executes a ferry task every 5 seconds to ferry the files on the Internet FTP server to the intranet FTP server. The JSON scheduled parsing task deployed in the intranet domain will scan the structured data directory of the intranet FTP server every 10 seconds. After finding the corresponding file, it will parse and assemble the corresponding SQL and execute it on the target database. The structured data synchronization is successful. The SHELL script scheduled task deployed in the intranet domain will execute the command every 10 seconds to check whether there is a corresponding file in the unstructured directory of the intranet FTP server. If there is, it will execute the RSYNC command and add the parameter "--delelte-after" to cut and delete the directory operation, cut the unstructured data to the file server in the intranet domain, and the unstructured data synchronization is completed.
[0091] This embodiment provides a method for synchronizing structured and unstructured data across networks, including: respectively building an Internet FTP server, an intranet FTP server, and a network gate, and configuring the network gate file synchronization rules; upon receiving a data synchronization instruction, saving the structured data and unstructured data of the Internet domain to the Internet FTP server respectively; ferrying the files of the Internet FTP server to the intranet FTP server based on the network gate file synchronization rules; scanning the structured data directory of the intranet FTP server based on the JSON scheduled parsing task and performing structured data synchronization; searching the unstructured data directory of the intranet FTP server based on the SHELL script scheduled task and performing unstructured data synchronization. In this embodiment, by using network gates, SYNC, FTP servers, SHELL scripts, and codes, synchronization of structured and unstructured data in different network environments is achieved, and the data synchronization problem is solved in the absence of necessary network environments and tools.
[0092] In addition, an embodiment of the present invention also proposes a storage medium, on which a cross-network structured and unstructured data synchronization program is stored. When the cross-network structured and unstructured data synchronization program is executed by a processor, the steps of the cross-network structured and unstructured data synchronization method described above are implemented.
[0093] Reference Figure 7 , Figure 7 It is a structural block diagram of an embodiment of a cross-network structured and unstructured data synchronization device of the present invention.
[0094] like Figure 7 As shown, the cross-network structured and unstructured data synchronization device includes:
[0095] FTP server building module 10, used to build an Internet FTP server, an intranet FTP server and a network gate, and configure network gate file synchronization rules;
[0096] A synchronization trigger module 20, for saving the structured data and the unstructured data in the Internet domain to the Internet FTP server respectively upon receiving a data synchronization instruction;
[0097] A cross-network ferry module 30, for ferrying files from the Internet FTP server to the intranet FTP server based on the network gate file synchronization rule;
[0098] The structured data synchronization module 40 is used to scan the structured data directory of the intranet FTP server and perform structured data synchronization based on the JSON scheduled parsing task;
[0099] The unstructured data synchronization module 50 is used to search the unstructured data directory of the intranet FTP server and perform unstructured data synchronization based on the SHELL script scheduled task.
[0100] This embodiment proposes a cross-network structured and unstructured data synchronization device. In this embodiment, by adopting a network gate, SYNC, FTP, SHELL script and code, synchronization of structured data and unstructured data is achieved in different network environments, providing a solution to the data synchronization problem in the absence of necessary network environment and tools.
[0101] It should be noted that the technical details that are not described in detail in the embodiment of the cross-network structured and unstructured data synchronization device can be found in the cross-network structured and unstructured data synchronization method described above provided in any embodiment of the present invention, and will not be repeated here.
[0102] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0103] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0104] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0105] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0107] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for synchronizing structured and unstructured data across networks, It is characterized in that include: Set up an Internet FTP server, an intranet FTP server, and a network gatekeeper, and configure the network gatekeeper file synchronization rules; Upon receiving the data synchronization instruction, the structured data and the unstructured data in the Internet domain are saved to the Internet FTP server respectively; Transferring files from the Internet FTP server to the Intranet FTP server based on the network gate file synchronization rules; Scan the structured data directory of the intranet FTP server based on the JSON scheduled parsing task and perform structured data synchronization; Based on the SHELL script scheduled task, the unstructured data directory of the intranet FTP server is searched and the unstructured data synchronization is performed.
2. The method according to claim 1, It is characterized in that The steps of respectively building an Internet FTP server, an intranet FTP server and a network gate, and configuring the network gate file synchronization rules include: Set up an Internet FTP server in the Internet domain; An intranet FTP server is built in the intranet domain; wherein the Internet domain and the intranet domain are network isolated; A network gate is built and network gate file synchronization rules are configured so that the Internet FTP server and the intranet FTP server can perform cross-network file transfer regularly based on the network gate file synchronization rules.
3. The method according to claim 1, It is characterized in that The method further comprises: A JAVA program is deployed in an unstructured data storage server in an Internet domain; wherein the JAVA program is used to provide an API for executing a SYNC command so as to synchronize files in the unstructured data storage server to an unstructured data directory in an Internet FTP server.
4. The method according to claim 1, It is characterized in that When receiving the data synchronization instruction, the structured data and the unstructured data in the Internet domain are saved to the Internet FTP server respectively, including: When the unstructured data synchronization demand is triggered, call the API to assemble the SYNC command; executing the SYNC command to copy the unstructured data file from the unstructured data storage server; Uploading the unstructured data file and the directory structure to the unstructured data directory of the Internet FTP server; When the structured data synchronization demand is triggered, a preset rule JSON file is generated and the preset rule JSON file is uploaded to the Internet FTP server.
5. The method according to claim 4, It is characterized in that The generating of the preset rule JSON file and uploading the preset rule JSON file to the Internet FTP server includes: saving the structured data to a database in the Internet domain; Convert the structured data of the database into a preset rule JSON file based on a preset code; wherein the preset rule JSON file includes a target table name, a synchronization field, and a field value; Upload the preset rule JSON file to the structured data directory of the Internet FTP server.
6. The method according to claim 1, It is characterized in that The step of transferring the file of the Internet FTP server to the Intranet FTP server based on the network gate file synchronization rule includes: The ferrying program based on the network gate executes a ferrying task once at a first preset time interval; wherein the ferrying task is to ferry the file of the Internet FTP server to the intranet FTP server based on the network gate file synchronization rule.
7. The method according to any one of claims 1 to 6, It is characterized in that The JSON-based scheduled parsing task scans the structured data directory of the intranet FTP server and performs structured data synchronization, including: Scanning the structured data directory of the intranet FTP server at a second preset time interval based on the JSON timing parsing task; When a target JSON file is scanned in the structured data directory, the target JSON file is parsed and SQL is assembled to obtain parsed data and save the parsed data to a file server in the intranet domain; After parsing is complete, the target JSON file of the intranet FTP server is deleted to complete the synchronization of structured data.
8. The method according to any one of claims 1 to 6, It is characterized in that The SHELL script-based scheduled task searches for the unstructured data directory of the intranet FTP server and performs unstructured data synchronization, including: Scanning the unstructured data directory of the intranet FTP server at a third preset time interval based on a SHELL script scheduled task; When a target unstructured data file is scanned in the unstructured data directory, an RSYNC command is executed and the target unstructured data file is cut to a file server in the intranet domain to complete the synchronization of the unstructured data.
9. A device for synchronizing structured and unstructured data across networks, It is characterized in that include: FTP server building module, used to build Internet FTP server, intranet FTP server and network gate, and configure network gate file synchronization rules; A synchronization trigger module, for saving the structured data and the unstructured data in the Internet domain to the Internet FTP server respectively upon receiving a data synchronization instruction; A cross-network ferry module, used for ferrying files from the Internet FTP server to the intranet FTP server based on the network gate file synchronization rule; A structured data synchronization module, used to scan the structured data directory of the intranet FTP server and perform structured data synchronization based on the JSON scheduled parsing task; The unstructured data synchronization module is used to search the unstructured data directory of the intranet FTP server based on the SHELL script scheduled task and perform unstructured data synchronization.
10. An electronic device, It is characterized in that The electronic device includes: a memory, a processor, and a cross-network structured and unstructured data synchronization program stored in the memory and executable on the processor, wherein the cross-network structured and unstructured data synchronization program is configured to implement the cross-network structured and unstructured data synchronization method as described in any one of claims 1 to 8.
