Data transmission method and device, equipment and storage medium

Abstract

The invention discloses a data transmission method, device and equipment and a storage medium, and relates to the technical field of distribution.The data transmission method comprises the steps that a sub-file from a data sender is received, and the sub-file is stored in a first file directory; decrypting the sub-file by adopting a preset encryption algorithm, and moving the decrypted sub-file to a second file directory; a file transmission message from a distributed file monitoring tool is received and responded, file merging is carried out on the sub-files in the second file directory based on the file transmission message, and a target file is generated; and when the number of the target files reaches a preset target number threshold value, starting a batch file processing job. By means of the technical scheme, thread resources are saved, and meanwhile the timeliness of batch file processing operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to a data transmission method, device, equipment and storage medium. Background Art

[0002] In a distributed system, there is a process of batch processing of a large number of files, and these files are usually not on the same physical server. When processing distributed batch files, in some business scenarios, it is necessary to wait for multiple files to arrive before allowing the process to start.

[0003] Currently, there are two ways to process distributed batch files. One is to create an independent thread. For example, scan the file directory once every second and determine whether the file directory contains all the necessary files for batch file processing. The batch processing job is started when the conditions are met. The second is to start the batch processing job service at regular intervals. When it is determined that the files are not complete, the processing is exited. When the files are complete, the business processing of the batch job is actually started. The former consumes memory resources unnecessarily, resulting in low utilization of system resources. The latter solution has a low efficiency due to the execution of batch jobs in multiple cycles. Summary of the invention

[0004] The present invention provides a data transmission method, device, equipment and storage medium, which reduce the consumption of memory resources and improve the efficiency of batch file processing.

[0005] According to one aspect of the present invention, a data transmission method is provided, which is applied to a data receiving party, and the method comprises:

[0006] Receiving a subfile from a data sender, and storing the subfile in a first file directory;

[0007] Decrypting the subfile using a preset symmetric encryption algorithm, and moving the decrypted subfile to a second file directory;

[0008] Receiving a file transfer message from a distributed file listening tool and responding to it, and merging the sub-files in the second file directory based on the file transfer message to generate a target file; wherein the target file is a file required by the data recipient to perform a batch file processing operation;

[0009] When the number of the target files reaches a preset target number threshold, a batch file processing operation is started.

[0010] According to another aspect of the present invention, a data transmission method is provided, which is applied to a data sender, and the method comprises:

[0011] The target file is divided into at least one sub-file, and the sub-files are encrypted by a preset symmetric encryption algorithm and then sent to the data receiver in a random order; wherein the naming information of the sub-files is determined according to the file name of the target file and the order in which the sub-files are divided;

[0012] When each sub-file is successfully sent, the data sender will notify the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver, so that the data receiver can merge the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

[0013] According to another aspect of the present invention, there is provided a data transmission device, which is applied to a data receiving party, and the device comprises:

[0014] A file receiving module, used for receiving a sub-file from a data sender, and storing the sub-file in a first file directory;

[0015] A file decryption module, used to decrypt the sub-file using a preset symmetric encryption algorithm, and move the decrypted sub-file to a second file directory;

[0016] A file merging module, configured to receive and respond to a file transfer message from a distributed file monitoring tool, and merge the sub-files in the second file directory based on the file transfer message to generate a target file; wherein the target file is a file required by a data recipient to perform a batch file processing operation;

[0017] The batch processing module is used to start a batch file processing job when the number of the target files reaches a preset target number threshold.

[0018] According to another aspect of the present invention, there is provided a data transmission device, which is applied to a data receiving party, and the device comprises:

[0019] A file sending module is used to divide the target file into at least one sub-file, and encrypt the sub-files using a preset symmetric encryption algorithm and then send them to the data recipient in a random order; wherein the naming information of the sub-files is determined according to the file name of the target file and the order in which the sub-files are divided;

[0020] The message sending module is used to notify the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver after each sub-file is successfully sent, so that the data receiver can merge the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data transmission method described in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data transmission method described in any embodiment of the present invention when executed.

[0026] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data transmission method according to any embodiment of the present invention is implemented.

[0027] The technical solution of the embodiment of the present invention uses a multi-level file directory to process the received sub-files to generate a target file, standardizes the file data processing flow, classifies and stores the sub-files at different processing stages, avoids data pollution, improves data processing efficiency, and merges the target files through file transfer messages, converting the imperceptible file arrival event into a perceptible, listenable, and processable file transfer message, thereby saving thread resources and improving the timeliness of batch file processing jobs.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flowchart of a data transmission method provided according to Embodiment 1 of the present invention;

[0031] Figure 2is a flowchart of a data transmission method provided according to Embodiment 2 of the present invention;

[0032] Figure 3 is a flowchart of a data transmission method provided according to Embodiment 3 of the present invention;

[0033] Figure 4 is a structural schematic diagram of a data transmission device provided according to Embodiment 4 of the present invention;

[0034] Figure 5 is a structural schematic diagram of a data transmission device provided according to Embodiment 5 of the present invention;

[0035] Figure 6 It is a structural schematic diagram of an electronic device for implementing the data transmission method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment 1

[0039] Figure 1 A flowchart of a data transmission method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a data receiver receives a file from a data sender. The method can be executed by a data transmission device. The data transmission device can be implemented in the form of hardware and / or software. The data transmission device can be configured in various general-purpose computing devices. Figure 1 As shown, the method includes:

[0040] S110: Receive a sub-file from a data sender, and store the sub-file in a first file directory.

[0041] The sub-file may refer to a sub-file of a file required by the data recipient for batch file processing. The first file directory may refer to a temporary directory for receiving files by the data recipient.

[0042] It should be noted that in the embodiment of the present invention, the data receiver and the data sender are data nodes in a distributed system, and the identities of the data sender and the data receiver can be interchanged. When a data node in a distributed system receives data, the data node is the data receiver; when a data node in a distributed system sends data, the data node is the data sender. In this distributed system, the encryption algorithm is pre-negotiated between the data nodes, and the reliability of the data transmission process can be ensured by the data encryption algorithm when data is transmitted between the data nodes. At the same time, the data receiving node can receive data transmitted from at least one data sending node. Optionally, the encryption algorithm can be adaptively set according to those skilled in the art, and is not specifically limited here.

[0043] Specifically, when the data receiving party receives a sub-file transmitted from at least one data sending party, the data receiving party first stores the sub-file in the first file directory.

[0044] S120: Decrypt the sub-file using a preset encryption algorithm, and move the decrypted sub-file to a second file directory.

[0045] The second file directory is used to store the decrypted received data.

[0046] In an optional embodiment of the present invention, the sub-file is decrypted using a preset encryption algorithm, and the decrypted sub-file is moved to a second file directory, and the method also includes: if a sub-file having the same naming information as the decrypted sub-file exists in the second file directory, the sub-file is replaced with the decrypted sub-file.

[0047] In the embodiment of the present invention, there may be subfiles with different versions sent multiple times, and the naming information of these subfiles is the same, but the file contents of the subfiles may have changed. Therefore, if a subfile with the same naming information as the newly decrypted subfile already exists in the second file directory, the subfile already existing in the second file directory is replaced with the newly decrypted subfile, and the newly received subfile shall prevail.

[0048] S130: Receive a file transfer message from the distributed file monitoring tool and respond to it, merge the sub-files in the second file directory based on the file transfer message, and generate a target file.

[0049] Among them, the distributed file monitoring tool can be a real-time data monitoring tool set up in the distributed system, which is used to process and analyze data flow data. When the distributed file monitoring tool monitors that a sub-file is successfully sent by the data sender, a file transfer message will be generated and sent to the file receiver of the sub-file. The file transfer message may include the file naming information, file summary information and sub-file quantity threshold of the sub-file; the target file can be the file required by the data receiver for batch file processing operations. Optionally, the target file required by the data receiver is generally stored in other data nodes in the distributed system.

[0050] It should be noted that during the file transmission process, in order to ensure data security, the data sender can split the target file into parts and then encrypt and send them. Therefore, after receiving the sub-files, the data receiver needs to merge the sub-files before obtaining the target file.

[0051] S140: When the number of target files reaches a preset target number threshold, a batch file processing job is started.

[0052] Specifically, when the data receiver monitors that the received target files reach the preset target number threshold required for the batch file processing job, the batch file processing job is started. Optionally, the preset target number threshold can be adaptively set according to those skilled in the art.

[0053] The technical solution of the embodiment of the present invention uses a multi-level file directory to process the received sub-files to generate a target file, standardizes the file data processing flow, classifies and stores the sub-files at different processing stages, avoids data pollution, improves data processing efficiency, and merges the target files through file transfer messages, converting the imperceptible file arrival event into a perceptible, listenable, and processable file transfer message, thereby saving thread resources and improving the timeliness of batch file processing jobs.

[0054] Embodiment 2

[0055] Figure 2 This is a flowchart of a data transmission method provided in the second embodiment of the present invention. This embodiment further refines the above embodiment and provides specific steps for receiving and responding to a file transfer message from a distributed file monitoring tool, merging sub-files in the second file directory based on the file transfer message, and generating a target file. It should be noted that for the undetailed parts of the embodiment of the present invention, please refer to the relevant descriptions of other embodiments, which will not be repeated here. Figure 2 As shown, the method includes:

[0056] S210: Receive a sub-file from a data sender, and store the sub-file in a first file directory.

[0057] S220: Decrypt the sub-file using a preset encryption algorithm, and move the decrypted sub-file to a second file directory.

[0058] S230: Traverse the sub-files in the second file directory according to the preceding naming information in the file naming information, determine a target sub-file, and move the target sub-file to the third file directory.

[0059] The sub-files stored in the third file directory are files to be merged and have the same pre-naming information. It should be noted that the naming information of the sub-files is composed of pre-naming information and post-naming information. The pre-naming information can be the file naming information of the parent file to which the sub-file belongs, and the post-naming information can be used to represent the segmentation order when the sub-files are segmented.

[0060] Specifically, after receiving the file transfer message sent by the distributed file monitoring tool, the data recipient can put the file transfer message into a message queue whose entry and exit rules are first-in-first-out; when the third file directory is empty, the data recipient can extract the file transfer message from the message queue in sequence, and traverse the sub-files in the second file directory according to the pre-naming information of the file naming information in the file transfer message, determine the sub-file in the second file directory that has the same pre-naming information as the target sub-file, and move the target sub-file to the third file directory. When the third file directory is not empty, if the pre-naming information in the file transfer message extracted by the data recipient from the message queue is different from the pre-naming information of the sub-file in the third file directory, the file transfer information is newly put into the message queue, and the file transfer information is extracted from the message queue again to ensure that the sub-files in the third file directory have the same pre-naming information.

[0061] In an optional implementation of the embodiment of the present invention, the sub-files in the second file directory are traversed according to the preceding naming information in the file naming information to determine the target sub-file, including: traversing the sub-files in the second file directory according to the preceding naming information in the file naming information to determine the candidate sub-files; verifying the candidate sub-files according to the file summary information in the file naming information, and if the verification passes, determining the candidate sub-file as the target sub-file.

[0062] The candidate sub-files are verified through the file summary information to determine the accuracy of the generated target file.

[0063] S240: When the number of files in the third file directory reaches a sub-file number threshold, the sub-files in the third file directory are merged according to the order of the post-naming information in the file naming information to generate a target file.

[0064] The sub-file number threshold may be used to characterize the number of sub-files after the target file is segmented. Optionally, the sub-file number threshold may be adaptively set according to those skilled in the art.

[0065] It should be noted that after the target file is generated, the target file can be saved in a file batch processing server set by the data recipient to prepare for the batch file processing job.

[0066] S250: When the number of target files reaches a preset target number threshold, a batch file processing job is started.

[0067] The technical solution of the embodiment of the present invention merges sub-files through the leading naming information and the trailing naming information in the file naming information to generate a target file, thereby ensuring the accuracy of the generated target file.

[0068] Embodiment 3

[0069] Figure 3 A flowchart of a data transmission method is provided for the third embodiment of the present invention. This embodiment is applicable to the case where a data sender sends file data to a data receiver. The method can be executed by a data transmission device. The data transmission device can be implemented in the form of hardware and / or software. The data transmission device can be configured in various general-purpose computing devices. Figure 3 As shown, the method includes:

[0070] S310, dividing the target file into at least one sub-file, and encrypting the sub-files using a preset encryption algorithm and then sending them to the data receiver in random order; wherein the naming information of the sub-files is determined according to the file name of the target file and the order in which the sub-files are divided.

[0071] S320. After each sub-file is successfully sent, the data sender notifies the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver, so that the data receiver merges the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

[0072] In an embodiment of the present invention, when a data sender in a distributed system sends a target file to a data receiver, the target file is divided into at least one sub-file, and the sub-file is encrypted using a preset encryption algorithm and then sent to the data receiver in random order to improve file transmission efficiency; when the data sender successfully sends the sub-file, the distributed file monitoring tool set in the distributed system will monitor the file transfer action, and generate a file transfer message for the sub-file and send it to the data receiver.

[0073] The technical solution of the embodiment of the present invention merges target files through file transfer messages, converts the imperceptible file arrival event into a perceptible, listenable and processable file transfer message, thereby saving thread resources and improving the timeliness of batch file processing jobs.

[0074] Embodiment 4

[0075] Figure 4 This is a schematic diagram of a data transmission device applied to a data receiving party provided in Embodiment 4 of the present invention. Figure 4 As shown, the device comprises:

[0076] The file receiving module 410 is used to receive the sub-file from the data sender and store the sub-file in the first file directory;

[0077] A file decryption module 420, configured to decrypt the sub-file using a preset symmetric encryption algorithm and move the decrypted sub-file to a second file directory;

[0078] The file merging module 430 is used to receive and respond to a file transfer message from a distributed file monitoring tool, and merge the sub-files in the second file directory based on the file transfer message to generate a target file; wherein the target file is a file required by the data recipient to perform a batch file processing operation;

[0079] The batch processing module 440 is used to start a batch file processing job when the number of the target files reaches a preset target number threshold.

[0080] The technical solution of the embodiment of the present invention uses a multi-level file directory to process the received sub-files to generate a target file, standardizes the file data processing flow, classifies and stores the sub-files at different processing stages, avoids data pollution, improves data processing efficiency, and merges the target files through file transfer messages, converting the imperceptible file arrival event into a perceptible, listenable, and processable file transfer message, thereby saving thread resources and improving the timeliness of batch file processing jobs.

[0081] Optionally, the file transfer message may include file naming information of the sub-file, file summary information, and a sub-file quantity threshold.

[0082] Optionally, the file merging module 430 includes:

[0083] a target sub-file determining unit, configured to traverse the sub-files in the second file directory according to the preceding naming information in the file naming information, determine the target sub-file, and move the target sub-file to the third file directory;

[0084] The target file generating unit is used to merge the sub-files in the third file directory according to the order of the post-naming information in the file naming information to generate the target file when the number of files in the third file directory reaches the sub-file number threshold.

[0085] Optionally, the sub-files stored in the third file directory may be files to be merged and have the same prefix naming information.

[0086] Optionally, the target sub-file determination unit can be specifically used to: traverse the sub-files in the second file directory according to the preceding naming information in the file naming information to determine a candidate sub-file; verify the candidate sub-file according to the file summary information in the file naming information, and if the verification passes, determine the candidate sub-file as the target sub-file.

[0087] Optionally, the leading naming information may be the file naming information of the parent file to which the child file belongs, and the trailing naming information may be used to indicate the segmentation order when the child file is segmented.

[0088] Optionally, the device further comprises:

[0089] The file replacement module is used to replace the sub-file with the decrypted sub-file if there is a sub-file with the same naming information as the decrypted sub-file in the second file directory.

[0090] The data transmission device applied to a data recipient provided in an embodiment of the present invention can execute the data transmission method applied to a data recipient provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0091] Embodiment 5

[0092] Figure 5 This is a schematic diagram of the structure of a data transmission device applied to a data sender provided in Embodiment 5 of the present invention. Figure 5 As shown, the device comprises:

[0093] The file sending module 510 is used to divide the target file into at least one sub-file, and encrypt the sub-file using a preset symmetric encryption algorithm and then send it to the data receiving party in a random order; wherein the naming information of the sub-file is determined according to the file name of the target file and the division order of the sub-file;

[0094] The message sending module 520 is used to notify the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver after each sub-file is successfully sent, so that the data receiver can merge the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

[0095] The technical solution of the embodiment of the present invention merges target files through file transfer messages, converts the imperceptible file arrival event into a perceptible, listenable and processable file transfer message, thereby saving thread resources and improving the timeliness of batch file processing jobs.

[0096] The data transmission device applied to the data sender provided in the embodiment of the present invention can execute the data transmission method applied to the data sender provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0097] Embodiment 6

[0098] Figure 6 A schematic diagram of an electronic device 610 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0099] like Figure 6As shown, the electronic device 610 includes at least one processor 611, and a memory connected to the at least one processor 611 in communication, such as a read-only memory (ROM) 612, a random access memory (RAM) 613, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 611 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 612 or the computer program loaded from the storage unit 618 to the random access memory (RAM) 613. In the RAM 613, various programs and data required for the operation of the electronic device 610 can also be stored. The processor 611, the ROM 612, and the RAM 613 are connected to each other via a bus 614. An input / output (I / O) interface 615 is also connected to the bus 614.

[0100] A number of components in the electronic device 610 are connected to the I / O interface 615, including: an input unit 616, such as a keyboard, a mouse, etc.; an output unit 617, such as various types of displays, speakers, etc.; a storage unit 618, such as a disk, an optical disk, etc.; and a communication unit 619, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 619 allows the electronic device 610 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The processor 611 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 611 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 611 performs the various methods and processes described above, such as a data transmission method.

[0102] In some embodiments, the data transmission method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 618. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 610 via the ROM 612 and / or the communication unit 619. When the computer program is loaded into the RAM 613 and executed by the processor 611, one or more steps of the data transmission method described above may be performed. Alternatively, in other embodiments, the processor 611 may be configured to perform the data transmission method in any other appropriate manner (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data transmission method, characterized in that: Applied to a data receiver, the method comprises: Receiving a subfile from a data sender, and storing the subfile in a first file directory; Decrypting the subfile using a preset encryption algorithm, and moving the decrypted subfile to a second file directory; Receiving a file transfer message from a distributed file listening tool and responding to it, and merging the sub-files in the second file directory based on the file transfer message to generate a target file; wherein the target file is a file required by the data recipient to perform a batch file processing operation; When the number of the target files reaches a preset target number threshold, a batch file processing operation is started.

2. The method according to claim 1, characterized in that in, The file transfer message includes file naming information of the sub-file, file summary information and a sub-file quantity threshold; Correspondingly, receiving a file transfer message from a distributed file monitoring tool and responding thereto, and merging sub-files in the second file directory based on the file transfer message to generate a target file, including: Traversing the sub-files in the second file directory according to the preceding naming information in the file naming information, determining a target sub-file, and moving the target sub-file to the third file directory; When the number of files in the third file directory reaches the sub-file number threshold, the sub-files in the third file directory are merged according to the order of the post-naming information in the file naming information to generate a target file.

3. The method according to claim 2, characterized in that The sub-files stored in the third file directory are files to be merged and have the same prefix naming information.

4. The method according to claim 2, characterized in that: Traversing the sub-files in the second file directory according to the preceding naming information in the file naming information to determine the target sub-file includes: Traversing the sub-files in the second file directory according to the preceding naming information in the file naming information to determine candidate sub-files; The candidate sub-file is verified according to the file summary information in the file naming information, and if the verification passes, the candidate sub-file is determined as the target sub-file.

5. The method according to claim 2, characterized in that: The leading naming information is the file naming information of the parent file to which the child file belongs, and the trailing naming information is used to indicate the segmentation order when the child file is segmented.

6. The method according to claim 1, characterized in that Decrypting the subfile using a preset encryption algorithm and moving the decrypted subfile to a second file directory also includes: If there is a sub-file in the second file directory that has the same naming information as the decrypted sub-file, the sub-file is replaced with the decrypted sub-file.

7. A data transmission method, characterized in that: Applied to a data sender, the method comprises: The target file is divided into at least one sub-file, and the sub-files are encrypted by a preset encryption algorithm and then sent to the data receiving party in a random order; wherein the naming information of the sub-files is determined according to the file name of the target file and the order in which the sub-files are divided; When each sub-file is successfully sent, the data sender will notify the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver, so that the data receiver can merge the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

8. A data transmission device, characterized in that: Applied to a data receiving party, the device comprises: A file receiving module, used for receiving a sub-file from a data sender, and storing the sub-file in a first file directory; A file decryption module, used to decrypt the sub-file using a preset symmetric encryption algorithm, and move the decrypted sub-file to a second file directory; A file merging module, configured to receive and respond to a file transfer message from a distributed file monitoring tool, and merge the sub-files in the second file directory based on the file transfer message to generate a target file; wherein the target file is a file required by a data recipient to perform a batch file processing operation; The batch processing module is used to start a batch file processing job when the number of the target files reaches a preset target number threshold.

9. A data transmission device, characterized in that: Applied to a data sender, the device comprises: A file sending module is used to divide the target file into at least one sub-file, and encrypt the sub-files using a preset symmetric encryption algorithm and then send them to the data recipient in a random order; wherein the naming information of the sub-files is determined according to the file name of the target file and the order in which the sub-files are divided; The message sending module is used to notify the distributed file monitoring tool to send a file transfer message of the sub-file to the data receiver after each sub-file is successfully sent, so that the data receiver can merge the target file according to the file transfer message; wherein the file transfer message includes the naming information of the sub-file, the file summary information and the file path information.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data transmission method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data transmission method according to any one of claims 1 to 7 when executed.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the data transmission method according to any one of claims 1 to 7 when executed by a processor.

Images