Data cross-network transmission method and device

Abstract

The present invention provides a data cross-network transmission method and device, which is applied to an external network processing device, and the method includes: collecting data to be processed, and storing the data to be processed in a message queue; classifying and packaging the data to be processed into multiple second data to be processed according to a preset number of items or a preset data size; when sending the second data to be processed, for each second data to be processed, performing a hash calculation on the second data to be processed, using the obtained hash value as the file name of the second data to be processed, and caching the second data to be processed in a distributed database Redis in the form of a file name as key and file content as value; using multithreading to ferry the second data to be processed to an internal network processing device; receiving a response file sent by the internal network processing device, and deleting the second data to be processed in Redis based on the response file. The above method improves transmission efficiency and ensures data integrity.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a method and device for cross-network data transmission. Background Art

[0002] In the information age, the Internet has been deeply integrated into all aspects of economic and social life, and network security threats have also penetrated into all levels of economic and social life, and the importance of network security has continued to increase. In the fields of public security and justice, due to their business characteristics, they have very high requirements for network security. Business data is not allowed to be stored in domains with lower confidentiality levels. However, most of the front-end devices for information collection are basically deployed in domains with relatively low confidentiality levels. How to transmit the data collected by the equipment to the application system safely, quickly and stably.

[0003] In the prior art, data is transferred from a low-density network domain to a high-density network domain by FTP file transfer. However, this method has low data transmission efficiency and cannot be perceived in the event of data loss. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for cross-network data transmission to achieve the purpose of improving data transmission efficiency and ensuring data integrity.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for transmitting data across a network, which is applied to an external network processing device. The method includes:

[0007] Collecting data to be processed and storing the data to be processed in a message queue;

[0008] Classify and package the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size;

[0009] When sending the second data to be processed, for each of the second data to be processed, a hash calculation is performed on the second data to be processed, the obtained hash value is used as the file name of the second data to be processed, and the second data to be processed is cached in the distributed database Redis in the form of the file name as key and the file content as value;

[0010] ferrying the second to-be-processed data to an intranet processing device using multithreading;

[0011] Receive a response file sent by the intranet processing device, and based on the response file, delete the second data to be processed in the Redis, wherein the response file is generated by the intranet processing device based on the second data to be processed and carries identification information of the second data to be processed.

[0012] Optionally, before collecting the data to be processed, the method further includes:

[0013] Capture unstructured data;

[0014] The unstructured data is encoded to obtain data to be processed.

[0015] Optionally, the using multithreading to transfer the second to-be-processed data to an intranet processing device includes:

[0016] Configure multiple Kafka partitions and set threads that match the number of Kafka partitions;

[0017] Use the multiple Kafka partitions and the multiple threads to ferry the second data to be processed to an intranet processing device.

[0018] Optionally, the receiving a response file sent by the intranet processing device, and deleting the second data to be processed in the Redis based on the response file, includes:

[0019] receiving a response file sent by the intranet processing device, and acquiring a file name of the second data to be processed based on the identification information of the second data to be processed;

[0020] Determine a key in the Redis that matches the file name, and determine second data to be processed corresponding to the key;

[0021] Delete the second data to be processed according to the key.

[0022] Optionally, also include:

[0023] If the response file sent by the intranet processing device is not received, query the cached data in the Redis periodically;

[0024] If the second data to be processed is found, the second data to be processed is resent to the intranet processing device until a response file sent by the intranet processing device is received.

[0025] A second aspect of an embodiment of the present invention discloses a method for transmitting data across a network, which is applied to an intranet processing device. The method includes:

[0026] Receive the second data to be processed ferrying from the external network processing device, wherein the second data to be processed is stored in the distributed database Redis in the form of a file name as a key and a file content as a value;

[0027] Calculating a hash value of the second data to be processed;

[0028] Verifying the hash value and the file name;

[0029] If the hash value and the file name are inconsistent, it is determined that the second data to be processed is lost, and the second data to be processed is discarded;

[0030] If the hash value is consistent with the file name, the response file of the second data to be processed is fed back to the external network processing device.

[0031] A third aspect of an embodiment of the present invention discloses a data cross-network transmission device, which is applied to an external network processing device. The device includes:

[0032] A collection and storage module, used for collecting data to be processed and storing the data to be processed in a message queue;

[0033] A classification and packaging module, used for classifying and packaging the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size;

[0034] A hash processing module, used for performing hash calculation on each of the second data to be processed when sending the second data to be processed, using the obtained hash value as the file name of the second data to be processed, and caching the second data to be processed in the distributed database Redis in the form of the file name as key and the file content as value;

[0035] A ferry module, used for ferrying the second to-be-processed data to an intranet processing device using multithreading;

[0036] A deletion module is used to receive a response file sent by the intranet processing device, and based on the response file, delete the second data to be processed in the Redis. The response file is generated by the intranet processing device based on the second data to be processed and carries identification information of the second data to be processed.

[0037] Optionally, before the acquisition and storage module is used to acquire the data to be processed, it is also used to:

[0038] Obtain unstructured data; encode the unstructured data to obtain data to be processed.

[0039] Optionally, the ferry module includes:

[0040] A configuration unit, used to configure multiple Kafka partitions and set threads consistent with the number of the Kafka partitions;

[0041] A ferry unit is used to ferry the second data to be processed to an intranet processing device using the multiple Kafka partitions and the multiple threads.

[0042] Optionally, the deletion module includes:

[0043] A receiving unit, configured to receive a response file sent by the intranet processing device, and obtain a file name of the second data to be processed based on the identification information of the second data to be processed;

[0044] A determination unit, configured to determine a key in the Redis that matches the file name, and determine second data to be processed corresponding to the key;

[0045] A deleting unit is used to delete the second data to be processed according to the key.

[0046] A data cross-network transmission method and device provided based on the above-mentioned embodiment of the present invention are applied to an external network processing device, and the method includes: collecting data to be processed, and storing the data to be processed in a message queue; classifying and packaging the data to be processed into multiple second data to be processed according to a preset number of items or a preset data size; when sending the second data to be processed, for each second data to be processed, performing a hash calculation on the second data to be processed, using the obtained hash value as the file name of the second data to be processed, and caching the second data to be processed in a distributed database Redis in the form of the file name as key and the file content as value; using multiple threads to ferry the second data to be processed to an intranet processing device; receiving a response file sent by the intranet processing device, and deleting the second data to be processed in Redis based on the response file, wherein the response file is generated by the intranet processing device based on the second data to be processed and carries identification information of the second data to be processed. In this solution, first, the data to be processed is classified and packaged to reduce the transmission frequency. Secondly, the second data to be processed is hashed and the hash value is used as the file name. The second data to be processed is cached with the hash value, and multi-threading is used to ferry the second data to be processed to the intranet processing device to improve transmission efficiency. After receiving the response file from the intranet processing device, the second data to be processed is deleted to ensure data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a flow chart of a method for cross-network data transmission provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a data ferry process provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a process for deleting second data to be processed in Redis based on a response file provided in an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of a flow chart of another method for transmitting data across networks provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a flow chart of another method for transmitting data across networks provided in an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure of a data cross-network transmission device provided by an embodiment of the present invention;

[0054] Figure 7 A schematic diagram of the structure of another device for transmitting data across networks provided by an embodiment of the present invention;

[0055] Figure 8 A schematic diagram of the structure of another device for cross-network data transmission provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0059] As can be seen from the background technology, the existing FTP file ferrying method has low data transmission efficiency and cannot be perceived in the event of data loss.

[0060] Therefore, an embodiment of the present invention provides a method and device for cross-network data transmission. In this scheme, first, the data to be processed is classified and packaged to reduce the transmission frequency. Secondly, the obtained second data to be processed is hashed, and the obtained hash value is used as the file name. The second data to be processed is cached with the hash value, and multi-threading is used to ferry the second data to be processed to the intranet processing device to improve transmission efficiency. After receiving the response file from the intranet processing device, the second data to be processed is deleted to ensure data integrity.

[0061] like Figure 1 , which is a flow chart of a method for cross-network data transmission provided by an embodiment of the present invention, and the method is applied to an external network processing device.

[0062] The method mainly includes the following steps:

[0063] Step S101: Collect data to be processed and store the data to be processed in a message queue.

[0064] In the specific implementation of step S101, when the external network processing device performs data processing, it needs to first collect the required data from the target device before performing subsequent data processing. That is, the external network processing device collects the data to be processed in the target device and stores the data to be processed in the message queue.

[0065] Optionally, before executing step S101 to collect the data to be processed, the following steps are also included:

[0066] Step S11: Obtain unstructured data.

[0067] In step S11, the unstructured data includes but is not limited to vehicle and face images generated by camera capture, and face photos and ID photos generated by person-ID comparison.

[0068] Step S12: Encode the unstructured data to obtain data to be processed.

[0069] Preferably, in a specific embodiment, Base64 encoding is performed on the unstructured data to obtain data to be processed.

[0070] Step S102: classify and package the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size.

[0071] In step S102, the preset number of entries and the preset data size can be flexibly configured.

[0072] Preferably, the preset number of entries may be 100, and the preset data size may be 200M.

[0073] In the specific implementation of step S102, the external network processing device classifies and packages the data to be processed according to a preset number of items or a preset data size to obtain a plurality of second data to be processed.

[0074] Preferably, the external network processing device classifies and packages the data to be processed according to 100 or 200M to obtain a plurality of second data to be processed.

[0075] Step S103: When sending the second data to be processed, for each second data to be processed, a hash calculation is performed on the second data to be processed, and the obtained hash value is used as the file name of the second data to be processed, and the second data to be processed is cached in the distributed database Redis in the form of file name as key and file content as value.

[0076] It should be noted that the distributed database (Remote Dictionary Server, Redis) is pre-established and is an open source log-type, Key-Value database that can be based on memory or persistence.

[0077] In the process of specifically implementing step S103, when the external network processing device sends the second data to be processed, it performs hash calculation on each second data to be processed to obtain a specific hash value, and uses the hash value as the file name of the second data to be processed. The second data to be processed is cached in the distributed database Redis in the form of the file name as the key and the file content as the value, that is, in the form of the hash value as the key and the file content as the value.

[0078] Step S104: Use multithreading to transfer the second data to be processed to the intranet processing device.

[0079] In the embodiment of the present invention, multithreading is used to perform FTP transfer on the second data to be processed.

[0080] Preferably, the external network processing device uses multi-threading to perform FTP transfer on the second data to be processed, and transfers it to the internal network processing device.

[0081] Optionally, the process of executing step S104 using multithreading to transfer the second data to be processed to the intranet processing device is as follows: Figure 2 FIG. 1 is a flow chart of a data ferry provided by an embodiment of the present invention, which mainly includes the following steps:

[0082] Step S201: configure multiple Kafka partitions, and set threads that are consistent with the number of Kafka partitions.

[0083] In the specific implementation of step S201, when the external network processing device consumes data using Kafka, it configures multiple Kafka partitions according to data consumption requirements and sets threads that are consistent with the number of Kafka partitions.

[0084] Step S202: Use multiple Kafka partitions and multiple threads to ferry the second data to be processed to the intranet processing device.

[0085] In the specific implementation of step S202, after completing the configuration of the Kafka partitions and threads, the external network processing device uses the configured multiple Kafka partitions and multiple threads to ferry the second data to be processed to the internal network processing device.

[0086] It is understandable that when the service corresponding to the external network processing device starts to consume data through Kafka, it can store data according to the configuration of multiple Kafka partitions. The number of threads is consistent with the number of Kafka partitions, and the file writing speed is improved by consuming data through multiple partitions and multiple threads.

[0087] Step S105: receiving a response file sent by the intranet processing device, and deleting the second data to be processed in Redis based on the response file.

[0088] In step S105, a response file is generated by the intranet processing device based on the second data to be processed, and carries identification information of the second data to be processed.

[0089] In the embodiment of the present invention, the response file refers to a response file that the internal network processing device automatically writes back after receiving the data content transferred from the external network processing device, which is equivalent to signing for the data, indicating that the internal network processing device has received the data sent by the external network processing device.

[0090] It should be noted that the external network processing device receives the response file sent by the internal network processing device, indicating that no data is lost during the ferrying process.

[0091] In the process of specifically implementing step S105, the external network processing device receives the response file sent by the internal network processing device. After receiving the response file, the identification information of the second data to be processed is determined based on the response file, and the second data to be processed corresponding to the identification information in Redis is deleted according to the identification information of the second data to be processed.

[0092] Optionally, executing step S105 to receive a response file sent by the intranet processing device, and deleting the second data to be processed in Redis based on the response file, such as Figure 3 As shown, a flow chart of deleting the second data to be processed in Redis based on a response file provided by an embodiment of the present invention mainly includes the following steps:

[0093] Step S301: receiving a response file sent by an intranet processing device, and obtaining a file name of the second data to be processed based on identification information of the second data to be processed.

[0094] In the specific implementation of step S301, the external network processing device receives the response file sent by the internal network processing device, determines the identification information of the second data to be processed based on the response file, and obtains the file name of the second data to be processed according to the identification information of the second data to be processed.

[0095] Step S302: Determine the key in Redis that matches the file name, and determine the second data to be processed corresponding to the key.

[0096] In the specific implementation of step S302, the external network processing device searches for a key in Redis that matches the file name of the second data to be processed, and determines the second data to be processed corresponding to the key.

[0097] Step S303: Delete the second data to be processed according to the key.

[0098] In the process of specifically implementing step S303, the external network processing device deletes the second data to be processed according to the key, that is, the external network processing device deletes the cache data corresponding to the second data to be processed in Redis according to the file name key, or the external network processing device uses the same file name key to delete the value data of the corresponding key.

[0099] A data cross-network transmission method provided based on an embodiment of the present invention is applied to an external network processing device, by collecting data to be processed and storing the data to be processed in a message queue; classifying and packaging the data to be processed into multiple second data to be processed according to a preset number of items or a preset data size; when sending the second data to be processed, for each second data to be processed, performing a hash calculation on the second data to be processed, using the obtained hash value as the file name of the second data to be processed, and caching the second data to be processed in a distributed database Redis in the form of a file name as a key and a file content as a value; using multiple threads to ferry the second data to be processed to an internal network processing device; receiving a response file sent by the internal network processing device, and deleting the second data to be processed in Redis based on the response file, wherein the response file is generated by the internal network processing device based on the second data to be processed and carries identification information of the second data to be processed. In this solution, first, the data to be processed is classified and packaged to reduce the transmission frequency. Secondly, the second data to be processed is hashed and the hash value is used as the file name. The second data to be processed is cached with the hash value, and multi-threading is used to ferry the second data to be processed to the intranet processing device to improve transmission efficiency. After receiving the response file from the intranet processing device, the second data to be processed is deleted to ensure data integrity.

[0100] Based on the above embodiment of the present invention, a data cross-network transmission method is provided, such as Figure 4 FIG. 1 is a flow chart of another method for transmitting data across networks provided by an embodiment of the present invention, the method mainly comprising the following steps:

[0101] Step S401: Collect data to be processed and store the data to be processed in a message queue.

[0102] Step S402: classify and package the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size.

[0103] Step S403: When sending the second data to be processed, for each second data to be processed, a hash calculation is performed on the second data to be processed, and the obtained hash value is used as the file name of the second data to be processed, and the second data to be processed is cached in the distributed database Redis in the form of the file name as the key and the file content as the value.

[0104] Step S404: Use multithreading to transfer the second data to be processed to the intranet processing device.

[0105] Step S405: Determine whether a response file sent by the intranet processing device is received, if so, execute step S406, if not, execute step S407.

[0106] Step S406: based on the response file, delete the second data to be processed in Redis.

[0107] It should be noted that the execution principle and process of the above steps S401 to S406 are the same as Figure 1 The execution principles and processes of steps S101 to S105 disclosed in are the same, which can be referred to and will not be repeated here.

[0108] Step S407: Periodically query the cached data in Redis.

[0109] In the specific implementation of step S407, if the external network processing device determines that the response file sent by the internal network processing device is not received, it will periodically query the cache data in Redis.

[0110] Step S408: Determine whether the second data to be processed is found. If so, execute step S409; if not, return to execute step S407.

[0111] Step S409: resending the second data to be processed to the intranet processing device until a response file sent by the intranet processing device is received.

[0112] In the process of implementing step S409, if the external network processing device determines that the second data to be processed is found, the second data to be processed is resent to the internal network processing device, and steps S403 to S404 are re-executed until a response file sent by the internal network processing device is received, and the operation is terminated.

[0113] That is to say, if the external network processing device determines that the second data to be processed is queried, the second data to be processed will be resent to the internal network processing device, and the hash calculation will be performed on each second data to be processed again. The obtained hash value will be used as the file name of the second data to be processed, and the second data to be processed will be cached in the distributed database Redis in the form of file name as key and file content as value, and the second data to be processed will be transferred to the internal network processing device using multiple threads until the response file sent by the internal network processing device is received, and the operation is terminated.

[0114] A data cross-network transmission method provided by an embodiment of the present invention, firstly, reduces the transmission frequency by classifying and packaging the data to be processed, and secondly, performs hash calculation on the obtained second data to be processed, and uses the obtained hash value as the file name, caches the second data to be processed by the hash value, and uses multi-threading to ferry the second data to be processed to an intranet processing device to improve transmission efficiency, deletes the second data to be processed after receiving a response file from the intranet processing device to ensure data integrity, and periodically queries the cached data in Redis when the response file from the intranet processing device is not received, and resends the second data to be processed to the intranet processing device after the second data to be processed is queried, until the response file sent by the intranet processing device is received, thereby preventing missing and loss during data transmission.

[0115] Based on the above embodiment of the present invention, a data cross-network transmission method is provided, such as Figure 5 , which is a flow chart of another method for cross-network data transmission provided by an embodiment of the present invention, and the method is applied to an intranet processing device.

[0116] The method mainly includes the following steps:

[0117] Step S501: receiving second data to be processed transferred by an external network processing device.

[0118] In step S501, the second data to be processed is stored in the distributed database Redis in the form of a file name as a key and a file content as a value.

[0119] In the specific implementation of step S501, when the external network processing device uses multi-threading to perform FTP transfer of the second data to be processed, the internal network processing device receives the second data to be processed transferred by the external network processing device.

[0120] It should be noted that the intranet processing device receives the second data to be processed ferried from the external network processing device, which can be understood as: writing a file in the ftp on one side and ferrying (or cutting) it to the ftp folder on the other side.

[0121] Step S502: Calculate the hash value of the second data to be processed.

[0122] In the process of specifically implementing step S502, the intranet processing device calculates a hash value for the received second data to be processed to obtain a specific hash value.

[0123] Step S503: Check the hash value and the file name. If the hash value and the file name are inconsistent, execute step S504. If the hash value and the file name are consistent, execute step S505.

[0124] Step S504: Determine that the second data to be processed is lost, and discard the second data to be processed.

[0125] In the specific implementation of step S504, the intranet processing device determines that the calculated hash value is inconsistent with the file name pre-stored in the distributed database Redis, then determines that the second data to be processed is lost in the process of ferrying data by the external network processing device, and discards the second data to be processed.

[0126] Step S505: Feedback the response file of the second data to be processed to the external network processing device.

[0127] In the process of implementing step S505, the intranet processing device determines that the calculated hash value is consistent with the file name pre-stored in the distributed database Redis, then generates a response file for the second data to be processed, and feeds back the response file of the second data to be processed to the external network processing device. In other words, the response information of the second data to be processed is written back to the external network processing device.

[0128] A data cross-network transmission method provided based on an embodiment of the present invention is applied to an intranet processing device. After receiving the second data to be processed transferred from the external network processing device, the hash value of the second data to be processed is calculated. When the hash value is inconsistent with a pre-stored file name, it is determined that the second data to be processed is lost and the second data to be processed is discarded. When the hash value is consistent with the pre-stored file name, a response file of the second data to be processed is fed back to the external network processing device, thereby ensuring data integrity.

[0129] The above-mentioned embodiment of the present invention Figure 1 Corresponding to the data cross-network transmission method shown in FIG. , the embodiment of the present invention also provides a data cross-network transmission device, such as Figure 6 As shown, the device is applied to an external network processing device, and the device includes: a collection and storage module 61, a classification and packaging module 62, a hash processing module 63, a ferry module 64 and a deletion module 65.

[0130] The collection and storage module 61 is used to collect the data to be processed and store the data to be processed in the message queue.

[0131] The classification and packaging module 62 is used to classify and package the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size.

[0132] The hash processing module 63 is used to perform hash calculation on each second data to be processed when sending the second data to be processed, use the obtained hash value as the file name of the second data to be processed, and cache the second data to be processed in the distributed database Redis in the form of file name as key and file content as value.

[0133] The ferry module 64 is used to ferry the second to-be-processed data to the intranet processing device using multiple threads.

[0134] The deletion module 65 is used to receive a response file sent by the intranet processing device, and delete the second data to be processed in Redis based on the response file.

[0135] The response file is generated by the intranet processing device based on the second data to be processed, and carries identification information of the second data to be processed.

[0136] Optional, based on the above Figure 6 The collection and storage module 61 shown, before being used to collect the data to be processed, is also used to:

[0137] Obtain unstructured data; encode the unstructured data to obtain data to be processed.

[0138] Optional, based on the above Figure 6 The ferry module 64 shown includes:

[0139] A configuration unit is used to configure multiple kafka partitions and set threads consistent with the number of the kafka partitions.

[0140] The ferry unit is used to ferry the second to-be-processed data to the intranet processing device by using multiple Kafka partitions and multiple threads.

[0141] Optional, based on the above Figure 6 The deletion module 65 shown includes:

[0142] The receiving unit is used to receive a response file sent by the intranet processing device, and obtain a file name of the second data to be processed based on the identification information of the second data to be processed.

[0143] The determination unit is used to determine the key matching the file name in Redis and determine the second data to be processed corresponding to the key.

[0144] The deleting unit is used to delete the second data to be processed according to the key.

[0145] It should be noted that the specific principles and execution processes of each module in the data cross-network transmission device disclosed in the above embodiment of the present invention are the same as those of the data cross-network transmission method implemented in the above embodiment of the present invention. Please refer to the corresponding parts of the data cross-network transmission method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0146] A data cross-network transmission device provided based on an embodiment of the present invention is applied to an external network processing device, which collects data to be processed and stores the data to be processed in a message queue; classifies and packages the data to be processed into multiple second data to be processed according to a preset number of items or a preset data size; when sending the second data to be processed, for each second data to be processed, performs a hash calculation on the second data to be processed, uses the obtained hash value as the file name of the second data to be processed, and caches the second data to be processed in a distributed database Redis in the form of a file name as a key and a file content as a value; uses multi-threading to ferry the second data to be processed to an internal network processing device; receives a response file sent by the internal network processing device, and deletes the second data to be processed in Redis based on the response file, wherein the response file is generated by the internal network processing device based on the second data to be processed and carries identification information of the second data to be processed. In this solution, first, the data to be processed is classified and packaged to reduce the transmission frequency. Secondly, the second data to be processed is hashed and the hash value is used as the file name. The second data to be processed is cached with the hash value, and multi-threading is used to ferry the second data to be processed to the intranet processing device to improve transmission efficiency. After receiving the response file from the intranet processing device, the second data to be processed is deleted to ensure data integrity.

[0147] Optional, based on the above Figure 6 The data cross-network transmission device shown is combined with Figure 6 ,like Figure 7 As shown, the data cross-network transmission device is further provided with a query module 66 and a retransmission module 67.

[0148] The query module 66 is used to query the cached data in Redis at regular intervals if the response file sent by the intranet processing device is not received.

[0149] The resending module 67 is used to resend the second data to be processed to the intranet processing device if the second data to be processed is found, until a response file sent by the intranet processing device is received.

[0150] A data cross-network transmission device provided based on an embodiment of the present invention, firstly, reduces the transmission frequency by classifying and packaging the data to be processed, and secondly, performs hash calculation on the obtained second data to be processed, and uses the obtained hash value as the file name, caches the second data to be processed by the hash value, and uses multi-threading to ferry the second data to be processed to an intranet processing device to improve transmission efficiency, deletes the second data to be processed after receiving a response file from the intranet processing device to ensure data integrity, and periodically queries the cached data in Redis when the response file from the intranet processing device is not received, and resends the second data to be processed to the intranet processing device after the second data to be processed is queried, until the response file sent by the intranet processing device is received, thereby preventing missing and loss during data transmission.

[0151] The above-mentioned embodiment of the present invention Figure 5 Corresponding to the data cross-network transmission method shown in FIG. , the embodiment of the present invention also provides another data cross-network transmission device, such as Figure 8 As shown, the device is applied to an intranet processing device, and the device includes: a receiving module 81, a calculating module 82, a checking module 83, a discarding module 84 and a feedback module 85.

[0152] The receiving module 81 is used to receive the second data to be processed transferred by the external network processing device.

[0153] The second data to be processed is stored in the distributed database Redis in the form of a file name as a key and a file content as a value.

[0154] The calculation module 82 is used to calculate the hash value of the second data to be processed.

[0155] The verification module 83 is used to verify the hash value and the file name. If the hash value and the file name are inconsistent, the discarding module 84 is executed. If the hash value and the file name are consistent, the feedback module 85 is executed.

[0156] The discarding module 84 is used to determine that the second data to be processed is lost, and discard the second data to be processed.

[0157] The feedback module 85 is used to feed back the response file of the second data to be processed to the external network processing device.

[0158] It should be noted that the specific principles and execution processes of each module in the data cross-network transmission device disclosed in the above embodiment of the present invention are the same as those of the data cross-network transmission method implemented in the above embodiment of the present invention. Please refer to the corresponding parts of the data cross-network transmission method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0159] A data cross-network transmission device provided based on an embodiment of the present invention is applied to an intranet processing device. After receiving the second data to be processed transferred from the external network processing device, the hash value of the second data to be processed is calculated. When the hash value is inconsistent with a pre-stored file name, it is determined that the second data to be processed is lost and the second data to be processed is discarded. When the hash value is consistent with the pre-stored file name, a response file of the second data to be processed is fed back to the external network processing device, thereby ensuring data integrity.

[0160] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0161] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0162] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for transmitting data across networks, characterized in that: Applied to an external network processing device, the method comprises: Collecting data to be processed and storing the data to be processed in a message queue; Classify and package the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size; When sending the second data to be processed, for each of the second data to be processed, a hash calculation is performed on the second data to be processed, the obtained hash value is used as the file name of the second data to be processed, and the second data to be processed is cached in the distributed database Redis in the form of the file name as key and the file content as value; ferrying the second to-be-processed data to an intranet processing device using multithreading; Receive a response file sent by the intranet processing device, and based on the response file, delete the second data to be processed in the Redis, wherein the response file is generated by the intranet processing device based on the second data to be processed and carries identification information of the second data to be processed; The method of using multiple threads to transfer the second to-be-processed data to an intranet processing device includes: Configure multiple Kafka partitions and set threads that match the number of Kafka partitions; Use the multiple Kafka partitions and the multiple threads to ferry the second data to be processed to an intranet processing device.

2. The method according to claim 1, characterized in that Before collecting the data to be processed, the method further includes: Capture unstructured data; The unstructured data is encoded to obtain data to be processed.

3. The method according to claim 1, characterized in that The receiving a response file sent by the intranet processing device, and deleting the second data to be processed in the Redis based on the response file, includes: receiving a response file sent by the intranet processing device, and acquiring a file name of the second data to be processed based on the identification information of the second data to be processed; Determine a key in the Redis that matches the file name, and determine second data to be processed corresponding to the key; Delete the second data to be processed according to the key.

4. The method according to claim 1, characterized in that: Also includes: If the response file sent by the intranet processing device is not received, query the cached data in the Redis periodically; If the second data to be processed is found, the second data to be processed is resent to the intranet processing device until a response file sent by the intranet processing device is received.

5. A method for transmitting data across networks, characterized in that: Applied to an intranet processing device, the method comprises: Receive the second data to be processed ferrying from the external network processing device, wherein the second data to be processed is stored in the distributed database Redis in the form of a file name as a key and file content as a value; Calculating a hash value of the second data to be processed; Verifying the hash value and the file name; If the hash value and the file name are inconsistent, it is determined that the second data to be processed is lost, and the second data to be processed is discarded; If the hash value and the file name are consistent, feeding back a response file of the second data to be processed to the external network processing device; Among them, the external network processing device is configured with multiple Kafka partitions and is provided with threads consistent with the number of the Kafka partitions; the second data to be processed is ferried to the internal network processing device through the multiple Kafka partitions and multiple threads of the external network processing device.

6. A data cross-network transmission device, characterized in that: Applied to an external network processing device, the device comprises: A collection and storage module, used for collecting data to be processed and storing the data to be processed in a message queue; A classification and packaging module, used for classifying and packaging the data to be processed into a plurality of second data to be processed according to a preset number of items or a preset data size; A hash processing module, used for performing hash calculation on each of the second data to be processed when sending the second data to be processed, using the obtained hash value as the file name of the second data to be processed, and caching the second data to be processed in the distributed database Redis in the form of the file name as key and the file content as value; A ferry module, used for ferrying the second to-be-processed data to an intranet processing device using multithreading; a deletion module, configured to receive a response file sent by the intranet processing device, and delete the second data to be processed in the Redis based on the response file, wherein the response file is generated by the intranet processing device based on the second data to be processed and carries identification information of the second data to be processed; The ferry module comprises: A configuration unit, used to configure multiple Kafka partitions and set threads consistent with the number of the Kafka partitions; A ferry unit is used to ferry the second data to be processed to an intranet processing device using the multiple Kafka partitions and the multiple threads.

7. The device according to claim 6, characterized in that Before the acquisition storage module is used to acquire the data to be processed, it is also used to: Obtain unstructured data; encode the unstructured data to obtain data to be processed.

8. The device according to claim 6, characterized in that The deletion module comprises: A receiving unit, configured to receive a response file sent by the intranet processing device, and obtain a file name of the second data to be processed based on the identification information of the second data to be processed; A determination unit, configured to determine a key in the Redis that matches the file name, and determine second data to be processed corresponding to the key; A deleting unit is used to delete the second data to be processed according to the key.

Images