Internal and external network offline file processing method and system

Abstract

The invention relates to the technical field of data transmission, in particular to an internal and external network offline file processing method and system. The method comprises the following steps: reading a leakage-proof mark file, and obtaining a target file from a boundary platform; verifying the target file according to the unique identifier in the anti-leakage marked file; if the verification is successful, analyzing and storing the target file; and if the verification fails, not analyzing or not storing the target file. The method provided by the invention can ensure the consistency of internal and external network data, can detect packet leakage in real time, and is high in accuracy of packet leakage detection.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission. More specifically, the present invention relates to a method and system for processing off-line files between an internal network and an external network. Background Art

[0002] In actual application scenarios, such as schools, enterprises or government departments, in order to ensure the internal data security, an internal local area network (generally referred to as the internal network) is usually established, and the internal network and external networks such as the Internet (abbreviation: external network) are not directly connected. When data transmission and exchange are carried out, generally, a boundary platform developed by a third party is used for transfer and storage, that is, the external network needs to transmit the input data to the boundary platform, and then the internal network pulls the data off-line from the boundary platform through a special channel. However, when files are transmitted between the internal network and the external network, the situation of inconsistent data between the internal network and the external network may occur due to network problems (such as sudden network interruption resulting in data omission), or failures of the boundary platform (such as incorrect forwarding or packet loss).

[0003] Therefore, how to ensure the consistency of data between the internal network and the external network is a technical problem that urgently needs to be solved at present. Summary of the Invention

[0004] To solve the above technical problem of inconsistent data between the internal network and the external network caused by network problems or boundary platform failures, etc., the present invention provides solutions in the following aspects.

[0005] In a first aspect, the present invention provides a method for processing off-line files between an internal network and an external network, which is applied to an external network terminal, and includes: reading a leak prevention marker file, and obtaining a target file from a boundary platform; verifying the target file according to a unique identifier in the leak prevention marker file; if the verification is successful, parsing and storing the target file; if the verification fails, not parsing or storing the target file.

[0006] Beneficial effects: By only parsing and storing the target files that pass the verification, the storage of incorrect and invalid data is avoided, thereby ensuring the consistency of data between the internal network and the external network; at the same time, the situation of occupying system resources by parsing and storing invalid target files is avoided, thereby improving resource utilization. Moreover, it can detect in real time and quickly whether there is a packet loss situation in the current target file.

[0007] Further, before verifying the target file, the method of the present invention further includes: determining whether the time recorded in the leak prevention marker file is later than the time stamp included in the file name of the current target file, and if so, skipping the current target file and processing the next target file.

[0008] Beneficial effects: By skipping the target files repeatedly sent by the boundary platform, it is possible to avoid the situation of repeated parsing and storage that consume system resources, thereby improving resource utilization and data transmission efficiency.

[0009] Further, when obtaining the target file from the boundary platform, the method of the present invention further includes: recording the time of requesting the target file from the boundary platform, and in response to the time being greater than a preset threshold, stopping requesting the target file from the boundary platform.

[0010] Beneficial effects: By not continuing to request outdated data packets, it is possible to ensure data security and avoid resource waste caused by long-term requests for outdated data packets, thereby affecting the problem of transmission efficiency.

[0011] Further, when obtaining the target file from the boundary platform, the method of the present invention further includes: judging whether the target file is an invalid file according to the time information included in the anti-leakage marker file. If so, stop requesting the target file from the boundary platform; if not, continue to request the target file from the boundary platform.

[0012] Further, the method of the present invention further includes: generating a parsing log, and the content of the parsing log includes: parsing time, parsing object, parsing content, and parsing result.

[0013] In a second aspect, the present invention provides an internal and external network offline file processing method, which is applied to an internal network terminal and includes: writing the data to be transmitted into a file to be transmitted according to a preset data rule to obtain a target file, and the file name of the target file has the same part as the file name of the previous target file; transmitting the target file to the boundary platform.

[0014] Beneficial effects: By setting that the file name of the target file has the same part as the file name of the previous target file, it is convenient for the external network terminal to reorganize the data according to the association between the file names, ensuring the accuracy of data reorganization.

[0015] Further, writing the data to be transmitted into a file to be transmitted according to a preset data rule includes: in response to the data to be transmitted needing to be divided into multiple parts for transmission, writing the data to be transmitted into multiple files to be transmitted in a specified order.

[0016] Further, the target file is in JSON format or XML format.

[0017] Further, the method of the present invention further includes: generating an anti-leakage marker file corresponding to the target file, and the anti-leakage marker file is in TXT format.

[0018] In a third aspect, the present invention provides an off-line file processing system for internal and external networks, including an external network terminal, an internal network terminal, and a boundary platform. The boundary platform is communicatively connected to both the external network terminal and the internal network terminal. Among them, the external network terminal is used to implement an off-line file processing method for internal and external networks according to any one of the first aspect, the internal network terminal is used to implement an off-line file processing method for internal and external networks according to any one of the second aspect, and the boundary platform is used to transmit data between the external network terminal and the internal network terminal.

[0019] The beneficial effects of the present invention are as follows: By only parsing and storing the target files that pass the verification, the consistency of the internal and external network data is ensured. Moreover, by not parsing and not storing invalid target files, the problem of the system resources being occupied by parsing and storing invalid files can be avoided, thereby improving the resource utilization rate and data transmission efficiency; by skipping the target files repeatedly sent by the boundary platform, the repeated parsing and storing operations of the same file can be avoided, thereby improving the resource utilization rate; by not requesting outdated file packages, the system resources can be saved and the response speed can be improved. By verifying the target files according to the anti-leakage marker files, the leak detection can be realized in real time and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understood. In the drawings, several embodiments of the present invention are shown by way of illustration and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 is a flowchart schematically showing an off-line file processing method for internal and external networks according to Embodiment 1 of the present invention; Figure 2 is a flowchart schematically showing an off-line file processing method for internal and external networks according to Embodiment 2 of the present invention; Figure 3 is a schematic diagram schematically showing an unpacking log according to Embodiment 2 of the present invention; Figure 4 is a block diagram schematically showing the structure of an off-line file processing system for internal and external networks according to Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] It should be noted that in the present invention, the intranet referred to is a local area network dedicated to public security, and its file information has a high level of confidentiality. In addition, the intranet terminal, the extranet terminal, and the boundary platform involved in the present invention form a communication network with the functions of intranet-extranet isolation and interaction.

[0024] Among them, the boundary platform is located between the intranet and the extranet, and refers to a third-party platform for data transmission between the extranet terminal and the intranet terminal. This boundary platform is equivalent to a data transfer station, which can achieve communication isolation between the intranet and the extranet, thereby ensuring the communication security of the intranet. The intranet terminal refers to the terminal devices set in the internal networks such as the public security system and the security system, which can be gateways, servers, etc., and this embodiment does not make specific limitations; the extranet terminal refers to the terminal devices configured in the external networks such as the Internet, which can be gateways, servers, personal computers, etc., and this embodiment does not make specific limitations. In the present invention, the intranet terminal transfers the target file to the boundary platform, and then the extranet terminal regularly obtains the target file from the boundary platform, thereby realizing the transmission of intranet-extranet data.

[0025] Embodiment 1 Figure 1 FIG. is a flowchart of a method for processing offline files between the intranet and the extranet provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where the intranet terminal is driven to package data into a target file package and upload it to a boundary platform developed by a third party. This method can be executed by a file transmission device based on the intranet terminal. This file transmission device can be implemented by software and / or hardware and can be configured in computer devices such as servers and personal computers. The method specifically includes the following steps: S101. Write the data to be transmitted into a target file according to a preset data rule to obtain the target file.

[0026] In this embodiment, the preset data rule is in XML format or JSON format, which is determined by the compilation environment.

[0027] Specifically, write the data to be transmitted into the file to be exchanged in XML format or JSON format to obtain the target file, that is, the file format of the target file is XML format or JSON format.

[0028] It should be noted that there is an association or identical part between the file name of the generated target file and the file name of the previous target file (i.e., a target file generated before the current target file). Specifically, the latter half of the file name of the current target file is the former half of the file name of the previous target file, that is, the former half of the file name of the current target file is the latter half of the file name of the next target file. For example, if the file name of the previous target file is "b-a.json", then the file name of the current target file can be "c-b.json", and the file name of the next target file can be "d-c.json". In this example, json is the file format. In actual applications, the file name of the target file can be in the form of "prreq20250306180222545-prreq20250306174144091.json". In an alternative embodiment, it can also be set such that the former half of the file name of the current target file is the latter half of the file name of the previous target file, and those skilled in the art can set it according to actual needs.

[0029] It should be noted that for the first generated target file, its file name can be set such that both the front and back parts are the same, for example, "a-a.json". By setting it in this way, it can be quickly determined which target file is the first to be transmitted, facilitating subsequent sorting and recombination, while ensuring the consistency of the format of the front and back parts of the file name.

[0030] It should be noted that the above situation is for the case where the data volume is small and only needs to be transmitted once. For data to be transmitted with a relatively large data volume, such as video files, audio files, etc., it may be necessary to split them into multiple parts for transmission to improve the transmission efficiency and real-time performance. Specifically, the data to be transmitted is written into multiple files to be transmitted in a specified order, thereby obtaining multiple target files. For any target file, there is an identical part between the file name of this target file and the two adjacent target files.

[0031] By generating the file name of the target file according to this naming rule, the order of each target file (data packet) can be clarified, enabling the external network terminal to perform verification and sorting based on the association between the file names of adjacent target files. Especially for data that needs to be transmitted in multiple packets, in this case, the data needs to be recombined in sequence to obtain complete and accurate data. In addition, this naming method can also determine whether there is a problem of packet loss / missing packets. If the unique identifier in the anti-missing packet marker file of the external network terminal is inconsistent or does not match the file name of the target file, there is a situation of missing packets. In addition, after the external network terminal receives all the files and sorts them, if it detects that there is no identical part between the file names of two files, it indicates that there is a problem of packet loss / missing packets between these two target files.

[0032] S102. Generate a leakage prevention marker file corresponding to the target file.

[0033] Specifically, record the first half of the file name of the currently generated target file in the leakage prevention marker file (the leakage prevention marker file generated by the intranet terminal is a packaged leakage prevention marker file), and store it locally. It should be noted that there is a one-to-one correspondence between the leakage prevention marker file and the target file, and the leakage prevention marker file is a TXT file.

[0034] S103. Transmit the target file to the border platform.

[0035] In this embodiment, the border platform is an FPT interaction platform. Therefore, the intranet terminal transmits the data packet to the border platform through the FPT protocol.

[0036] Embodiment 2 Figure 2 The flowchart of an internal and external network offline file processing method provided by the second embodiment of the present invention. This embodiment is applicable to the situation where the external network terminal is driven to receive and parse the target file from the border platform developed by a third party. This method can be executed by a data processing device based on the external network terminal. The data processing device can be implemented by software and / or hardware and can be configured in a computer device. The method specifically includes the following steps: S201. Read the leakage prevention marker file and obtain the target file from the border platform.

[0037] Before obtaining the target file, user authentication can also be performed to ensure data security. When obtaining the target file, obtain the target file from the border platform according to the specified path. In this embodiment, the border platform is an FPT interaction platform. Further, read the leakage prevention marker file locally, and the leakage prevention marker file contains a unique identifier. It should be noted that the leakage prevention marker file of the external network terminal is a parsed leakage prevention marker file.

[0038] S202. Verify the target file according to the unique identifier in the leakage prevention marker file; if the verification is successful, parse and store the target file; if the verification fails, do not parse or store the target file.

[0039] In one embodiment, the verification process specifically includes: determining whether the unique identifier in the leakage prevention marker file meets the preset leakage prevention rules. If it meets, parse and store the target file; if it does not meet, do not parse or store the target file.

[0040] Specifically, if the unique identifier in the anti-leakage marker file is the same as the second half of the file name of the target file, it is determined that the target file meets the anti-leakage rule (indicating no missing packages), and the target file is parsed and stored; if the unique identifier in the anti-leakage marker file is different from the second half of the file name of the target file, it is determined that the target file does not meet the anti-leakage rule (missing packages occur), and the target file is not parsed or stored.

[0041] Specifically, when the external network terminal performs the first verification, it determines whether the first and second parts of the file name of the target file are the same. If they are the same, it is parsed, and the first half of the file name of the target file (which is the same as the second half of the file name of the next target file) is recorded in the anti-leakage marker file, that is, the first half of the file name of the current target file is recorded as the unique identifier in the anti-leakage marker file. When verifying the next target file, the anti-leakage marker file generated based on the file name of the previous target file is read, and then it is determined whether the unique identifier in the anti-leakage marker file is the same as the second half of the file name of the next target file. If so, it is parsed and stored; if not, it is not parsed or stored. According to the unique identifier in the anti-leakage marker file of the present invention, it can be determined whether there are missing packages in the target file, and the real-time performance is high. Since there is an association between the generated anti-leakage marker file and the next target file, the accuracy of missing package detection is high.

[0042] By only parsing and storing the target files that pass the verification, it is possible to avoid the situation where inconsistent internal and external network data is caused by parsing and storing invalid data, that is, to avoid problems such as missing packages and leakage caused by network problems or boundary platform failures. Then the external network terminal continues to parse and store the files with missing packages and leakage, resulting in inconsistent stored files and the files of the internal network terminal. At the same time, it can also avoid the problem of occupying system resources by parsing and storing invalid target files, thereby improving resource utilization and the efficiency of internal and external network data transmission.

[0043] Further, before performing the anti-leakage rule verification, it also includes: determining whether the time recorded in the anti-leakage marker file is later than the timestamp included in the second half of the target file. If so, skip the target file (i.e., do not parse or store the target file) and process the next target file; if it is not later than the time recorded in the target file, then parse and store the target file. It can be understood that if the FTP interaction platform fails, for example, the target file has been sent to the external network terminal, but due to certain reasons (such as network latency, fluctuations, judgment anomalies, etc.), the target file is resent to the external network terminal. After receiving the target file, the external network terminal will repeat operations such as parsing and storing, thus affecting the efficiency of data transmission and storage space. When the time recorded in the anti-leakage marker file is later than the timestamp included in the second half of the target file, it indicates that the boundary platform has resent the target file. At this time, by not parsing and storing the repeatedly sent target file, it is possible to avoid occupying additional system resources and improve resource utilization and the efficiency of data transmission between the internal and external networks.

[0044] In one embodiment, for the anti-leakage marker file with failed recognition, the target file can be requested and obtained from the FPT interaction platform again, further ensuring the consistency of the internal and external network data and correspondingly reducing the situation of packet loss / leakage. By monitoring whether there is a situation of packet loss / leakage, if there is a situation of packet loss / leakage, the file or data packet is automatically requested from the FPT platform again, further realizing the consistency and integrity of the internal and external network data.

[0045] Further, the data obtained after parsing is written into structured / unstructured data. For the data that needs to be transmitted in multiple parts, after receiving the target file, the external network terminal temporarily stores these target files in the cache, then sorts the target files according to the association between the file names of the target files, and then reorganizes the data obtained by parsing in order according to the original format to obtain the complete data.

[0046] In one embodiment, when requesting to obtain the target file from the boundary platform, the method of the present invention also includes: recording the time of requesting the target file from the boundary platform. If this time exceeds the preset threshold, stop requesting the target file from the boundary platform; if this time does not exceed the preset threshold, continue to request to obtain the target file from the boundary platform. By stopping the request for the target file with an overly long request time (i.e., not receiving outdated data packets), it is possible to avoid the resources occupied by invalid target files, improve the response speed and resource utilization rate, and thus improve the efficiency of data transmission.

[0047] In another embodiment, it is also possible to determine whether the target file is an invalid file according to the time information included in the file name of the anti-leakage marking file. Specifically, if the difference between the time recorded in the anti-leakage marking file and the current time exceeds a specified value, then the target file is an invalid file / outdated file, and the acquisition of the target file from the boundary platform is stopped; if the difference between the time in the anti-leakage marking file and the current time does not exceed the specified value, then the target file is a normal file / valid file, and the acquisition of the target file from the boundary platform continues.

[0048] In one embodiment, it is also possible to generate a parsing log / unpacking log. As Figure 3 shown, the content of the parsing log may include the parsing time, parsing result, parsing object, parsing process, parsing content, etc. In an optional embodiment, it may also include: the packet loss rate (which can be determined by counting the target files that fail the verification). In an optional embodiment, the packet loss rate can also be determined according to the ratio of the difference between the expected number of download requests and the number of times the anti-leakage marking file verification is successful to the expected number of download requests.

[0049] In summary, the method of the present invention does not parse and store target files with packet loss / leakage situations, avoiding the problem of inconsistent internal and external network data caused by parsing and storing invalid target files, thus ensuring the consistency and integrity of internal and external network data; by not parsing and storing outdated target files and invalid files, it is possible to avoid occupying additional resources, thereby improving resource utilization and transmission efficiency. Moreover, the data of the internal and external networks are generated separately and do not affect each other.

[0050] Figure 4 is a schematic block diagram showing the structure of an internal and external network offline file processing system according to an embodiment of the present invention.

[0051] In a third aspect, the present invention also provides an internal and external network offline file processing system. As Figure 4 shown, the internal and external network offline file processing system includes: an external network terminal, an internal network terminal, and a boundary platform. The boundary platform is communicatively connected to both the external network terminal and the internal network terminal. Among them, the external network terminal is used to implement an internal and external network offline file processing method according to any one of the first aspects, the internal network terminal is used to implement an internal and external network offline file processing method according to any one of the second aspects, and the boundary platform is used to transmit the data between the external network terminal and the internal network terminal.

[0052] In the description of this specification, the meaning of "a plurality" is at least two, such as two, three or more, etc., unless otherwise specifically defined. In addition, the step division of the above method is only for clear description. When implemented, it can be combined into one step or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included.

[0053] Although this specification has shown and described multiple embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will think of many changes, alterations and alternative ways without departing from the spirit and scope of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in the practice of the present invention.

Claims

1. A method for processing offline files on the internal and external networks, characterized in that, Applied to an external network terminal, including: Read the anti-leakage marker file and obtain the target file from the boundary platform; Verify the target file according to the unique identifier in the anti-leakage marker file; If the verification is successful, parse and store the target file; if the verification fails, do not parse or store the target file.

2. The off-line file processing method for internal and external networks according to claim 1, wherein Before verifying the target file, it also includes: Judge whether the time recorded in the anti-leakage marker file is later than the timestamp included in the file name of the current target file. If so, skip the current target file and process the next target file.

3. The offline file processing method for internal and external networks according to claim 1, characterized in that When obtaining the target file from the boundary platform, it also includes: record the time of requesting the target file from the boundary platform, and stop requesting the target file from the boundary platform in response to the time being greater than the preset threshold.

4. The offline file processing method for internal and external networks according to claim 1, characterized in that When obtaining the target file from the boundary platform, it also includes: judge whether the target file is an invalid file according to the time information included in the anti-leakage marker file. If so, stop requesting to obtain the target file from the boundary platform; if not, continue to request to obtain the target file from the boundary platform.

5. The offline file processing method for internal and external networks according to claim 1, characterized in that, It also includes: Generate a parsing log, and the content of the parsing log includes: parsing time, parsing object, parsing content, and parsing result.

6. A method for processing offline files on the internal and external networks, characterized in that, Applied to an internal network terminal, including: Write the data to be transmitted into a file to be transmitted according to the preset data rule to obtain a target file, and the file name of the target file has the same part as the file name of the previous target file; Transmit the target file to the boundary platform.

7. The off-line file processing method for internal and external networks according to claim 6, wherein Writing the data to be transmitted into a file to be transmitted according to the preset data rule includes: in response to the data to be transmitted needing to be divided into multiple parts for transmission, writing the data to be transmitted into multiple files to be transmitted in the specified order.

8. The off-line file processing method for internal and external networks according to claim 1, characterized in that, The target file is in JSON format or XML format.

9. The offline file processing method for internal and external networks according to claim 1, wherein It also includes: Generate an anti-leakage marker file corresponding to the target file, and the anti-leakage marker file is in TXT format.

10. An offline file processing system for internal and external networks, characterized in that, It includes an external network terminal, an internal network terminal, and a boundary platform. The boundary platform is communicatively connected to both the external network terminal and the internal network terminal. Among them, the external network terminal is used to implement an off-line file processing method for internal and external networks according to any one of claims 1-5, the internal network terminal is used to implement an off-line file processing method for internal and external networks according to any one of claims 6-9, and the boundary platform is used to transmit the data of the external network terminal and the internal network terminal.

Images