File processing method and device, equipment, medium and program product

Abstract

The invention discloses a file processing method and device, equipment, a medium and a program product. The method comprises the following steps: through a public layer container, when it is determined that a file collection time point is reached, sequentially accessing a first disk and a second disk until a first file sent by an upstream application is obtained; splitting the first file into a plurality of first sub-files through the public layer container, and storing each first sub-file into a first disk and a second disk; and reading a corresponding target first sub-file in the first disk or the second disk through each node layer container according to the file name of each first sub-file and the serial number of the node layer container, and storing the target first sub-file in a node layer database connected with the node layer container. According to the technical scheme, the method is suitable for the field of the distributed technology, the existing mode that the host core processes the files in batches can be replaced, processing such as cross-node file interaction and file splitting is achieved, and active-active processing of the distributed file system can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to a file processing method, device, equipment, medium and program product. Background Art

[0002] Distributed file processing is a file processing method that uses multiple computing nodes to work together to process files. The computing nodes are interconnected through a network and work together to complete file splitting, merging, sorting, deduplication and other operations.

[0003] In the prior art, a host core batch processing method and an open platform batch processing method are generally used to implement distributed batch file processing.

[0004] However, with the requirements of the transformation of domestic information technology, the host core batch processing method will gradually be phased out, and the open platform batch processing method generally relies on a specified database. When the database is unstable due to environmental problems and other factors, it is easy to cause batch job processing exceptions. At present, there is a lack of unified technical solutions for the sharding data of distributed databases and the aggregation of data transferred by file merging. Summary of the invention

[0005] The present invention provides a file processing method, device, equipment, medium and program product, which can replace the existing host core batch processing mode of files, realize cross-node file interaction and file splitting and other processing, and can realize dual-active processing of distributed file system.

[0006] According to one aspect of the present invention, a file processing method is provided, which is executed by a common layer container and each node layer container in a distributed application in cooperation, and includes:

[0007] Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained;

[0008] Splitting the first file into a plurality of first sub-files through a common layer container, and storing each of the first sub-files in a first disk and a second disk;

[0009] Through each node layer container, according to the file name of each first sub-file and the node layer container number, the corresponding target first sub-file is read in the first disk or the second disk, and the target first sub-file is stored in the node layer database connected to the node layer container.

[0010] Optionally, the distributed application further includes a first file server and a second file server, and the first disk and the second disk are mounted to the first path of the first file server and the second file server respectively;

[0011] The method further comprises:

[0012] Whenever a file sent by an upstream application is received through the first file server and the second file server, the files are stored in the first disk and the second disk respectively;

[0013] Wherein, through the common layer container, whenever it is determined that the file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, including:

[0014] Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk is accessed to determine whether there is a first file to be obtained in the first disk;

[0015] If so, the first file in the first disk is read; if not, the second disk is accessed and the first file in the second disk is read.

[0016] The advantage of this setting is that by using two disks to mount to the same file receiving server and each layer of container respectively, the distributed file system can be processed in an active-active manner. If an upstream node fails, it is guaranteed that at least one disk in the application has files stored in it, thus preventing the application from having a single point of failure due to a file receiving server or a distributed file system disk failure, and providing the best active-active solution for the application.

[0017] Optionally, the first disk is mounted to the second path in the public layer container and each node layer container, and the second disk is mounted to the third path in the public layer container and each node layer container;

[0018] Splitting the first file into a plurality of first sub-files through a common layer container, and storing each of the first sub-files in a first disk and a second disk, including:

[0019] Through the public layer container, the customer code field of each data row in the first file is read respectively, and the shard number corresponding to the customer code field is calculated by using a hash algorithm;

[0020] By using the common layer container, each data row is stored in a designated first sub-file according to the shard number of each data row, and a file name of each first sub-file is generated according to the file name of the first file and the shard number of the data row stored in each first sub-file;

[0021] Each first sub-file is stored in the second path and the third path respectively through the public layer container, so that each node layer container can share each first sub-file through the second path and the third path.

[0022] The advantage of this setting is that each node layer container can read the first sub-file under the second path and can also read the sub-file under the third path, thereby realizing data sharing between the public layer container and each node layer container, and being able to realize active-active processing of the distributed file system, thus ensuring the data security of the split files.

[0023] Optionally, each node layer container and each node layer database has a number respectively, and the node layer container is connected to the node layer database with the same number;

[0024] Reading the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number through each node layer container includes:

[0025] Through each node layer container, according to the file name of each first sub-file, the shard number of the data row in each first sub-file is determined, the first sub-file whose data row shard number is the same as the node layer container number is determined as the target first sub-file, and the target first sub-file corresponding to each node layer container is read in the second path or the third path.

[0026] The advantage of this setting is that a clear correspondence is formed between the container and the database through numbering. By selecting the target first sub-file according to the shard number, the system can reasonably distribute the data in different node layer databases. The distributed data storage method can avoid data concentration in a few storage locations and reduce hot data problems.

[0027] Optionally, the file processing method further includes:

[0028] When it is determined to send the second file to the downstream application through each node layer container, each second sub-file is read in each node layer database respectively connected to each node layer container, and stored in the first disk and the second disk;

[0029] Reading each second sub-file from the first disk or the second disk through the common layer container, and merging each second sub-file to generate a second file;

[0030] The second file is stored in the first path of the first file server and the second file server through the common layer container, so that the first file server and the second file server can read the second file in their respective first paths and send it to the downstream application.

[0031] The advantages of this setting are: it can also realize the file distribution of distributed applications, enrich the cross-node file operation form of the distributed file system, simplify the file interaction process, and realize the smooth migration of the host core batch processing method and the open platform batch processing method to the distributed file system.

[0032] Optionally, the distributed application also includes a distributed batch job scheduling component;

[0033] The file processing method further includes:

[0034] The distributed batch job scheduling component sends a file processing command to the common layer container and each node layer container, so that the common layer container and each node layer container perform the file processing operation according to the file processing command.

[0035] The advantage of this setting is that it provides a centralized way to manage file processing tasks. Both the public layer container and the node layer container operate according to the commands issued by the distributed batch job scheduling component, which helps to ensure that all containers process files according to unified rules and order, avoiding task confusion and conflicts.

[0036] According to another aspect of the present invention, there is provided a file processing device, which is executed by a common layer container and each node layer container in a distributed application in cooperation with each other, and includes:

[0037] A first file acquisition module is used to access the first disk and the second disk in sequence through the common layer container whenever it is determined that a file acquisition time point has arrived, until the first file sent by the upstream application is acquired;

[0038] A first file splitting module, used for splitting the first file into a plurality of first sub-files through a common layer container, and storing each first sub-file in a first disk and a second disk;

[0039] The sub-file storage module is used to read the corresponding target first sub-file in the first disk or the second disk through each node layer container according to the file name of each first sub-file and the node layer container number, and store the target first sub-file in the node layer database connected to the node layer container.

[0040] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0041] at least one processor; and

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file processing method described in any embodiment of the present invention.

[0044] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file processing method described in any embodiment of the present invention when executed.

[0045] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the file processing method described in any embodiment of the present invention is implemented.

[0046] The technical solution of the embodiment of the present invention utilizes a common layer container, and whenever it is determined that a file collection time point has been reached, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, and the first file is split into multiple first sub-files by utilizing the common layer container, and each first sub-file is stored in the first disk and the second disk, and each node layer container is utilized to read the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number, and the target first sub-file is stored in a node layer database connected to the node layer container. This method can replace the existing host core batch processing file method, and provides a safe and stable distributed file processing method, which can realize batch file processing across nodes, and can realize active-active processing of distributed file systems, avoid single point problems, and ensure data security of distributed applications.

[0047] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a flowchart of a file processing method provided according to Embodiment 1 of the present invention;

[0050] Figure 2 is a structural diagram of a distributed application provided according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of the structure of another file processing device provided according to Embodiment 2 of the present invention;

[0052] Figure 4 is a structural schematic diagram of a file processing device provided according to Embodiment 3 of the present invention;

[0053] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the file processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] Embodiment 1

[0057] Figure 1 This is a flowchart of a file processing method provided in the first embodiment of the present invention. This embodiment is applicable to the case where files sent by upstream applications are split and stored in segments. The method can be executed by a file processing device, which can be implemented in the form of hardware and / or software and can generally be configured in a distributed application with data processing functions. Figure 1 As shown, the method includes:

[0058] S110. Whenever it is determined that a file collection time point has arrived, the first disk and the second disk are accessed in sequence through the common layer container until the first file sent by the upstream application is obtained.

[0059] Figure 2 FIG. 1 is a schematic diagram of an optional distributed application structure. Figure 2As shown, two file receiving servers, two disks, a common layer (public layer) container, n agr layer (node ​​layer) containers, a Pub database connected to the common layer container, an agr (node ​​layer) database connected to each node layer container, and a distributed batch job scheduling component are set in the distributed application.

[0060] Optionally, the distributed application includes a public layer container, multiple node layer containers, a first file server, a second file server, a first disk and a second disk, and the first disk and the second disk are mounted to the first path of the first file server and the second file server respectively.

[0061] like Figure 2 As shown, the first file server and the second file server may be Figure 2 In the file receiving server 1 and the file receiving server 2, the first disk and the second disk can be Figure 2 Disk 1 and Disk 2 in the .

[0062] Optionally, when the upstream application sends the first file to the distributed application, the upstream application will send the first file to the first file server and the second file server at the same time. The first file server and the second file server are in a master-backup relationship with each other, and the first disk and the second disk are in a master-backup relationship with each other. After the first file server and the second file server receive the first file, they will respectively send the first file to the disk mounted on the file receiving server.

[0063] The advantage of this setting is that by using two disks to mount to the same file receiving server and each layer of container respectively, the distributed file system can be processed in an active-active manner. If an upstream node fails, it is guaranteed that at least one disk in the application has files stored in it, thus preventing the application from having a single point of failure due to a file receiving server or a distributed file system disk failure, and providing the best active-active solution for the application.

[0064] Optionally, the distributed application also includes a distributed batch job scheduling component;

[0065] The file processing method further includes: sending a file processing command to the common layer container and each node layer container through the distributed batch job scheduling component, so that the common layer container and each node layer container perform the file processing operation according to the file processing command.

[0066] Optionally, the distributed batch job scheduling component can send file processing commands to the public layer container and each node layer container according to the user's scheduling requirements, so that each container performs a corresponding operation. For example, the public layer container can access the disk at a specified time point to obtain the first file, and the node layer container can split the first file, etc. This is only an exemplary explanation.

[0067] Optionally, when the public layer container receives the collection instruction of the first file sent by the distributed batch job scheduling component, the first disk is first accessed through the public layer container. The first disk may be Figure 2 Disk 1 or Disk 2 is not limited to this. If the first file is not obtained in the first disk within the specified time range, the second disk is accessed to obtain the first file in the second disk; if the first file can be obtained in the first disk, there is no need to access the second disk.

[0068] S120: Split the first file into a plurality of first sub-files through a common layer container, and store each of the first sub-files in the first disk and the second disk.

[0069] Optionally, the purpose of splitting the first file into multiple first sub-files is to send each first sub-file to a node layer database for storage, so as to achieve file splitting and distributed storage in distributed applications.

[0070] Optionally, the public layer container may split the first file into first sub-files corresponding to the node layer databases according to a specified hash algorithm based on the database sharding situation in the distributed application, that is, the number of node layer databases.

[0071] In an optional example, through the common layer container, according to the guest code in the first file, the field of the guest code in each row of the first file is retrieved using a specified hash algorithm, and according to the guest code of each data row, the shard number to which each data row belongs is calculated. The shard number may correspond to the number of each node layer container, and each data row is placed in the first sub-file of the specified shard number. For example, there are 128 node layer containers and 128 node layer databases. Each node layer container and its corresponding node layer database have a unique number. Sub-files with shard numbers of 1-128 are created, and each sub-file corresponds to a node layer database.

[0072] S130. Through each node layer container, according to the file name of each first sub-file and the node layer container number, read the corresponding target first sub-file in the first disk or the second disk, and store the target first sub-file in the node layer database connected to the node layer container.

[0073] Optionally, each node layer container and each node layer database has a number respectively, and the node layer container is connected to the node layer database with the same number.

[0074] Optionally, the shard number of the first sub-file can be stored in a specified position in the file name of the first sub-file. Through each node layer container, the specified position of each first sub-file can be read to obtain the shard number of each first sub-file, and then the first sub-file corresponding to the shard number and the node layer container number is stored in the node layer database connected to the node layer container.

[0075] Optionally, the first disk is mounted to the second path in the public layer container and each node layer container, and the second disk is mounted to the third path in the public layer container and each node layer container.

[0076] It can be understood that the first disk and the second disk are mounted under the second path and the third path in the public layer container respectively. After the public layer container is split to obtain multiple first sub-files, the first sub-files can be stored in the second path and the third path at the same time. This operation is equivalent to storing each first sub-file in the first disk and the second disk respectively. The node layer container only needs to access the second path or the third path to share the first sub-file.

[0077] The advantage of this setting is that each node layer container can read the first sub-file under the second path and can also read the sub-file under the third path, thereby realizing data sharing between the public layer container and each node layer container, and being able to realize active-active processing of the distributed file system, thus ensuring the data security of the split files.

[0078] The technical solution of the embodiment of the present invention utilizes a common layer container, and whenever it is determined that a file collection time point has been reached, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, and the first file is split into multiple first sub-files by utilizing the common layer container, and each first sub-file is stored in the first disk and the second disk, and each node layer container is utilized to read the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number, and the target first sub-file is stored in a node layer database connected to the node layer container. This method can replace the existing host core batch processing file method, and provides a safe and stable distributed file processing method, which can realize batch file processing across nodes, and can realize active-active processing of distributed file systems, avoid single point problems, and ensure data security of distributed applications.

[0079] Embodiment 2

[0080] Figure 3This is a flowchart of a file processing method provided by Embodiment 2 of the present invention. This embodiment specifically describes the file processing method based on the above embodiment. Figure 3 As shown, the method includes:

[0081] S210. Whenever a file sent by an upstream application is received through the first file server and the second file server, the files are stored in the first disk and the second disk respectively.

[0082] S220. Whenever it is determined that a file collection time point has arrived, the first disk and the second disk are accessed in sequence through the common layer container until the first file sent by the upstream application is obtained.

[0083] Wherein, through the common layer container, whenever it is determined that the file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, which may include:

[0084] Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk is accessed to determine whether there is a first file to be obtained in the first disk;

[0085] If so, the first file in the first disk is read; if not, the second disk is accessed and the first file in the second disk is read.

[0086] S230. Read the customer code field of each data row in the first file through the common layer container, and calculate the shard number corresponding to the customer code field by using a hash algorithm.

[0087] Optionally, the customer code field is stored in a specified position in each data row, and the customer code field can be encoded using a hash algorithm to obtain a shard number corresponding to each data row.

[0088] S240. Through the common layer container, store each data row in a designated first sub-file according to the shard number of each data row, and generate a file name for each first sub-file according to the file name of the first file and the shard number of the data row stored in each first sub-file.

[0089] Optionally, a fragment number may be added at a specified position in the file name of the first file to generate a file name for each first sub-file.

[0090] In an optional implementation, the file name of the first file may be in the form of: F-XXXX-PARA-2023-01-01.BIN. After the first file is split, the file name of the first sub-file with fragment number 28 may be in the form of: F-XXXX-PARA-2023-01-01-028.BIN. This is only an example and does not limit the file name of the first sub-file.

[0091] S250 , storing each first sub-file in the second path and the third path respectively through the common layer container, so that each node layer container can share each first sub-file through the second path and the third path.

[0092] S260. Determine the shard number of the data row in each first subfile according to the file name of each first subfile through each node layer container, determine the first subfile whose data row shard number is the same as the node layer container number as the target first subfile, and read the target first subfiles corresponding to each node layer container in the second path or the third path.

[0093] The technical solution of the embodiment of the present invention utilizes a common layer container, and whenever it is determined that a file collection time point has been reached, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, and the first file is split into multiple first sub-files by utilizing the common layer container, and each first sub-file is stored in the first disk and the second disk, and each node layer container is utilized to read the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number, and the target first sub-file is stored in a node layer database connected to the node layer container. This method can replace the existing host core batch processing file method, and provides a safe and stable distributed file processing method, which can realize batch file processing across nodes, and can realize active-active processing of distributed file systems, avoid single point problems, and ensure data security of distributed applications.

[0094] Optionally, the file processing method further includes:

[0095] When it is determined to send the second file to the downstream application through each node layer container, each second sub-file is read in each node layer database respectively connected to each node layer container, and stored in the first disk and the second disk;

[0096] Reading each second sub-file from the first disk or the second disk through the common layer container, and merging each second sub-file to generate a second file;

[0097] The second file is stored in the first path of the first file server and the second file server through the common layer container, so that the first file server and the second file server can read the second file in their respective first paths and send it to the downstream application.

[0098] It is understandable that distributed applications not only need to receive files sent by upstream applications, but also need to send files to downstream applications.

[0099] Optionally, the second file may refer to a file that the distributed application needs to send to the downstream application. According to the above embodiment, the distributed application stores different sub-files of a file through multiple node layer databases. Therefore, if the second file is to be sent to the downstream application, it is necessary to first aggregate and generate the second file in the distributed application.

[0100] Optionally, the first file server and the second file server can be used as Figure 2 The file receiving server shown can also be used as a file sending server to send files to downstream applications.

[0101] Optionally, the second sub-file may refer to each sub-file after the second file is split, and the second sub-files may be merged to obtain the second file.

[0102] Optionally, the first file server and the second file server may respectively send the second file to two file servers in the downstream application through a file transfer tool, so as to implement active-active processing of the downstream application in the distributed file system.

[0103] The advantages of this setting are: it can also realize the file distribution of distributed applications, enrich the cross-node file operation form of the distributed file system, simplify the file interaction process, and realize the smooth migration of the host core batch processing method and the open platform batch processing method to the distributed file system.

[0104] Embodiment 3

[0105] Figure 4 This is a schematic diagram of the structure of a file processing device provided by Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a first file acquisition module 310, a first file splitting module 320 and a sub-file storage module 330.

[0106] The first file acquisition module 310 is used to access the first disk and the second disk in sequence through the common layer container whenever it is determined that a file acquisition time point has arrived, until the first file sent by the upstream application is acquired.

[0107] The first file splitting module 320 is used to split the first file into multiple first sub-files through a common layer container, and store each first sub-file in the first disk and the second disk.

[0108] The sub-file storage module 330 is used to read the corresponding target first sub-file in the first disk or the second disk through each node layer container according to the file name of each first sub-file and the node layer container number, and store the target first sub-file in the node layer database connected to the node layer container.

[0109] The technical solution of the embodiment of the present invention utilizes a common layer container, and whenever it is determined that a file collection time point has been reached, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, and the first file is split into multiple first sub-files by utilizing the common layer container, and each first sub-file is stored in the first disk and the second disk, and each node layer container is utilized to read the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number, and the target first sub-file is stored in a node layer database connected to the node layer container. This method can replace the existing host core batch processing file method, and provides a safe and stable distributed file processing method, which can realize batch file processing across nodes, and can realize active-active processing of distributed file systems, avoid single point problems, and ensure data security of distributed applications.

[0110] On the basis of the above embodiments, the distributed application further includes a first file server and a second file server, and the first disk and the second disk are mounted to the first path of the first file server and the second file server respectively;

[0111] It also includes a file receiving module, which is used to store files sent by the upstream application to the first disk and the second disk respectively through the first file server and the second file server whenever receiving the files;

[0112] The first file acquisition module 310 may be specifically used for:

[0113] Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk is accessed to determine whether there is a first file to be obtained in the first disk;

[0114] If so, the first file in the first disk is read; if not, the second disk is accessed and the first file in the second disk is read.

[0115] On the basis of the above embodiments, the first disk is mounted to the second path in the public layer container and each node layer container, and the second disk is mounted to the third path in the public layer container and each node layer container;

[0116] The first file splitting module 320 may be specifically used for:

[0117] Through the public layer container, the customer code field of each data row in the first file is read respectively, and the shard number corresponding to the customer code field is calculated by using a hash algorithm;

[0118] By using the common layer container, each data row is stored in a designated first sub-file according to the shard number of each data row, and a file name of each first sub-file is generated according to the file name of the first file and the shard number of the data row stored in each first sub-file;

[0119] Each first sub-file is stored in the second path and the third path respectively through the public layer container, so that each node layer container can share each first sub-file through the second path and the third path.

[0120] On the basis of the above embodiments, each node layer container and each node layer database has a number respectively, and the node layer container is connected to the node layer database with the same number;

[0121] The sub-file storage module 330 may be specifically used for:

[0122] Through each node layer container, according to the file name of each first sub-file, the shard number of the data row in each first sub-file is determined, the first sub-file whose data row shard number is the same as the node layer container number is determined as the target first sub-file, and the target first sub-file corresponding to each node layer container is read in the second path or the third path.

[0123] Based on the above embodiments, a file sending module may also be included, which is used to:

[0124] When it is determined to send the second file to the downstream application through each node layer container, each second sub-file is read in each node layer database respectively connected to each node layer container, and stored in the first disk and the second disk;

[0125] Reading each second sub-file from the first disk or the second disk through the common layer container, and merging each second sub-file to generate a second file;

[0126] The second file is stored in the first path of the first file server and the second file server through the common layer container, so that the first file server and the second file server can read the second file in their respective first paths and send it to the downstream application.

[0127] Based on the above embodiments, the distributed application further includes a distributed batch job scheduling component;

[0128] It also includes a command sending module, which is used to send file processing commands to the public layer container and each node layer container through the distributed batch job scheduling component, so that the public layer container and each node layer container perform file processing operations according to the file processing commands.

[0129] The file processing device provided in the embodiment of the present invention can execute the file processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0130] Embodiment 4

[0131] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0132] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0133] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the file processing method described in the embodiment of the present invention. That is:

[0135] Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained;

[0136] Splitting the first file into a plurality of first sub-files through a common layer container, and storing each of the first sub-files in a first disk and a second disk;

[0137] Through each node layer container, according to the file name of each first sub-file and the node layer container number, the corresponding target first sub-file is read in the first disk or the second disk, and the target first sub-file is stored in the node layer database connected to the node layer container.

[0138] In some embodiments, the file processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the file processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the file processing method in any other appropriate manner (e.g., by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0141] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0144] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A file processing method, executed by a common layer container and each node layer container in a distributed application, characterized in that: include: Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained; Splitting the first file into a plurality of first sub-files through a common layer container, and storing each of the first sub-files in a first disk and a second disk; Through each node layer container, according to the file name of each first sub-file and the node layer container number, the corresponding target first sub-file is read in the first disk or the second disk, and the target first sub-file is stored in the node layer database connected to the node layer container.

2. The method according to claim 1, characterized in that The distributed application further includes a first file server and a second file server, and the first disk and the second disk are mounted to the first path of the first file server and the second file server respectively; The method further comprises: Whenever a file sent by an upstream application is received through the first file server and the second file server, the files are stored in the first disk and the second disk respectively; Wherein, through the common layer container, whenever it is determined that the file collection time point has arrived, the first disk and the second disk are accessed in sequence until the first file sent by the upstream application is obtained, including: Through the common layer container, whenever it is determined that a file collection time point has arrived, the first disk is accessed to determine whether there is a first file to be obtained in the first disk; If so, the first file in the first disk is read; if not, the second disk is accessed and the first file in the second disk is read.

3. The method according to claim 1, characterized in that The first disk is mounted to the second path in the public layer container and each node layer container, and the second disk is mounted to the third path in the public layer container and each node layer container; Splitting the first file into a plurality of first sub-files through a common layer container, and storing each of the first sub-files in a first disk and a second disk, including: Through the public layer container, the customer code field of each data row in the first file is read respectively, and the shard number corresponding to the customer code field is calculated by using a hash algorithm; By using the common layer container, each data row is stored in a designated first sub-file according to the shard number of each data row, and a file name of each first sub-file is generated according to the file name of the first file and the shard number of the data row stored in each first sub-file; Each first sub-file is stored in the second path and the third path respectively through the public layer container, so that each node layer container can share each first sub-file through the second path and the third path.

4. The method according to claim 3, characterized in that Each node layer container and each node layer database has a number respectively, and the node layer container is connected to the node layer database with the same number; Reading the corresponding target first sub-file in the first disk or the second disk according to the file name of each first sub-file and the node layer container number through each node layer container includes: Through each node layer container, according to the file name of each first sub-file, the shard number of the data row in each first sub-file is determined, the first sub-file whose data row shard number is the same as the node layer container number is determined as the target first sub-file, and the target first sub-file corresponding to each node layer container is read in the second path or the third path.

5. The method according to claim 2, characterized in that: Also includes: When it is determined to send the second file to the downstream application through each node layer container, each second sub-file is read in each node layer database respectively connected to each node layer container, and stored in the first disk and the second disk; Reading each second sub-file from the first disk or the second disk through the common layer container, and merging each second sub-file to generate a second file; The second file is stored in the first path of the first file server and the second file server through the common layer container, so that the first file server and the second file server can read the second file in their respective first paths and send it to the downstream application.

6. The method according to any one of claims 1 to 5, characterized in that The distributed application also includes a distributed batch job scheduling component; The file processing method further includes: sending a file processing command to the common layer container and each node layer container through the distributed batch job scheduling component, so that the common layer container and each node layer container perform the file processing operation according to the file processing command.

7. A file processing device, executed by the cooperation of a common layer container and each node layer container in a distributed application, characterized in that: include: A first file acquisition module is used to access the first disk and the second disk in sequence through the common layer container whenever it is determined that a file acquisition time point has arrived, until the first file sent by the upstream application is acquired; A first file splitting module, used for splitting the first file into a plurality of first sub-files through a common layer container, and storing each first sub-file in a first disk and a second disk; The sub-file storage module is used to read the corresponding target first sub-file in the first disk or the second disk through each node layer container according to the file name of each first sub-file and the node layer container number, and store the target first sub-file in the node layer database connected to the node layer container.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file processing method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the file processing method according to any one of claims 1 to 6.

Images