Differential package generation method, target management node, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a differential package generation method, a target management node, electronic equipment, a storage medium and a program product, the method is applied to the target management node, and the method comprises the following steps: obtaining a source package and a target package used for generating a differential package; segmenting the source packet into a plurality of source files, and segmenting the target packet into a plurality of target files; determining a corresponding sub-task file group according to at least part of the target file, the sub-task file group corresponding to the target file comprising the target file and a corresponding source file; each sub-task file group is sent to a corresponding working node, so that each working node generates a corresponding first differential file according to a source file and a target file included in the received sub-task file group, the first differential file is sent to a target management node, and each working node is connected with the target management node; and generating a difference package according to at least part of the first difference file. According to the scheme, the efficiency of generating the differential package is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a differential packet generation method, a target management node, an electronic device, a storage medium, and a program product. Background Art

[0002] When manufacturers update the firmware or software of their products, they usually update based on differential packages in order to reduce the download time and traffic consumption of the products.

[0003] Currently, differential packages are generally generated in the manufacturer's server through a single-machine generation method. For example, during the period when the vehicle manufacturer uniformly upgrades the vehicle system, considering the limited processing power of the vehicle manufacturer's server, the server can generate patch packages for each part of the files in the source package and target package used to generate the differential package, and then generate a differential package based on the generated multiple patch packages.

[0004] However, multiple patch packages are generated sequentially, which results in a long time required to generate the multiple patch packages, thereby reducing the efficiency of generating differential packages. Summary of the invention

[0005] In view of this, embodiments of the present application provide a differential packet generation method, a target management node, an electronic device, a storage medium and a program product to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a differential package generation method is provided, which is applied to a target management node, and the method includes: obtaining a source package and a target package for generating a differential package; dividing the source package into multiple source files, and dividing the target package into multiple target files; determining a corresponding subtask file group based on at least part of the target file, wherein the subtask file group corresponding to the target file includes the target file and a corresponding source file; sending each of the subtask file groups to a corresponding working node, so that each of the working nodes generates a corresponding first differential file based on the source files and target files included in the received subtask file group, and sending the first differential file to the target management node, wherein each of the working nodes is connected to the target management node; generating a differential package based on at least part of the first differential file.

[0007] According to a second aspect of an embodiment of the present application, a target management node is provided, comprising: an acquisition unit for acquiring a source package and a target package for generating a differential package; a segmentation unit for segmenting the source package into multiple source files, and segmenting the target package into multiple target files; a determination unit for determining a corresponding subtask file group based on at least part of the target file, wherein the subtask file group corresponding to the target file includes the target file and a corresponding source file; a sending unit for sending each of the subtask file groups to a corresponding working node, so that each of the working nodes generates a corresponding first differential file based on the received source files and target files included in the subtask file group, and sends the first differential file to the target management node, wherein each of the working nodes is connected to the target management node; a generation unit for generating a differential package based on at least part of the first differential file.

[0008] According to the third aspect of an embodiment of the present application, there is provided an electronic device comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method of the first aspect described above.

[0009] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored, and the program is executed by a processor to perform the method of the first aspect.

[0010] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to execute the method of the first aspect described above.

[0011] According to the differential package generation scheme provided by the embodiment of the present application, the target management node can obtain the source package and the target package for generating the differential package, and then divide the source package into multiple source files, and divide the target package into multiple target files, and then determine the corresponding subtask file group according to at least part of the target files, and then send each subtask file group to the corresponding working node respectively, so that each working node generates the corresponding first differential file according to the source file and the target file included in the received subtask file group, and sends the first differential file to the target management node, and the target management node can generate the differential package according to at least part of the first differential file. Thus, through the setting of the target management node and the working node, it is possible to send the subtask file groups corresponding to at least some different target files to different working nodes, so that at least some different working nodes can simultaneously process the received subtask file groups to generate the corresponding first differential files, and then, the generation of at least part of the first differential files is parallel, which saves the time of generating the first differential file and improves the efficiency of generating the differential package. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is a flow chart of a differential packet generation method according to an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a target management node according to an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] Differential Packet Generation Method

[0017] An embodiment of the present application provides a differential packet generation method, which is applied to a target management node, where the target management node is any management node among multiple management nodes, each management node is connected to multiple working nodes, and both the management node and the working node can be cloud containers. For example, both the management node and the working node can be containers deployed in a cloud platform managed by Kubernetes (k8s).

[0018] The relationship between the central processing unit (CPU) core and memory included in the management node may be different from the relationship between the CPU core and memory included in the working node. For example, when deploying management nodes and working nodes in the cloud platform, the management node may deploy a1 CPU cores and b1 gigabytes (GB) of memory, b1≥2a1, and the working node may deploy a2 CPU cores and b2 gigabytes (GB) of memory, b2≥4a2 and b2>9.1K+3, where K is the data volume threshold below, and the 9.1K in "9.1K+3" is used to represent the estimated maximum memory used in the process of running the differential algorithm in the working node, and the 3 in "9.1K+3" is used to represent the estimated maximum memory used by other system processes in the working node.

[0019] When a working node goes online, it will automatically write information such as the maximum available memory and current available memory of its own Internet Protocol Address (IP address) into the working node resource table in the database. The database is a database to which both management nodes and working nodes have access permissions. In addition, all management nodes and working nodes will mount the directory (such as " / mnt") to the same cloud disk (such as a network attached storage (NAS) cloud disk), so that the cloud disk becomes a shared cloud disk for each management node and each working node.

[0020] The differential packet generation method is described in detail below through multiple embodiments.

[0021] Figure 1 FIG. 1 is a flow chart of a differential packet generation method according to an embodiment of the present application. Figure 1 As shown, the differential packet generation method includes the following steps:

[0022] Step 101: Obtain a source packet and a target packet for generating a differential packet.

[0023] The source packet is the data packet before updating, and the target packet is the data packet after updating.

[0024] In a specific implementation, the user can upload the source package and the target package in zip format to the shared cloud disk. The target management node can decompress the source package to the first subdirectory (for example, the " / mnt / ${task_id} / source_file" directory), and decompress the target package to the second subdirectory (for example, the " / mnt / ${task_id} / source_file" directory), so that the target management node can obtain the source package and the target package. Optionally, the target management node can generate a globally unique task identifier task_id for the source package and the target package to ensure the traceability and uniqueness of the task.

[0025] Step 102: split the source package into multiple source files, and split the target package into multiple target files.

[0026] Step 103: Determine a corresponding subtask file group according to at least part of the target file.

[0027] The subtask file group corresponding to the target file includes the target file and a corresponding source file.

[0028] Step 104: Send each subtask file group to a corresponding working node, so that each working node generates a corresponding first differential file according to the source file and target file included in the received subtask file group, and sends the first differential file to the target management node.

[0029] Among them, each working node is connected to the target management node.

[0030] Step 105: Generate a differential package based on at least part of the first differential file.

[0031] In an embodiment of the present application, the target management node can obtain a source package and a target package for generating a differential package, then divide the source package into a plurality of source files, and divide the target package into a plurality of target files, and then determine the corresponding subtask file group according to at least part of the target files, and then send each subtask file group to the corresponding working node respectively, so that each working node generates a corresponding first differential file according to the source file and the target file included in the received subtask file group, and sends the first differential file to the target management node, and the target management node can generate a differential package according to at least part of the first differential file. Thus, through the setting of the target management node and the working node, it is possible to send the subtask file groups corresponding to at least some different target files to different working nodes, so that at least some different working nodes can simultaneously process the received subtask file groups to generate the corresponding first differential files, and then, the generation of at least part of the first differential files is parallel, which saves the time of generating the first differential files and improves the efficiency of generating differential packages.

[0032] In a possible implementation, the above step 102 includes the following specific processing: dividing the source package into M source files, wherein the data volume of each source file is less than or equal to the data volume threshold, M≥2; dividing the target package into N target files, wherein the data volume of each target file is less than or equal to the data volume threshold, N≥2.

[0033] Among them, the data volume threshold can be set according to actual needs, for example, 1GB or 2GB, etc., and the embodiments of the present application are not limited to this.

[0034] In a specific embodiment, after the source package is stored in the first directory of the target management node, the target management node can determine the files included in the source package whose data volume is less than or equal to the data volume threshold as source files, and split the files included in the source package whose data volume is greater than the data volume threshold into multiple source files whose data volume is less than or equal to the data volume threshold, so as to obtain M source files; after the target package is stored in the second directory of the target management node, the target management node can determine the files included in the target package whose data volume is less than or equal to the data volume threshold as target files, and split the files included in the target package whose data volume is greater than the data volume threshold into multiple target files whose data volume is less than or equal to the data volume threshold, so as to obtain N target files.

[0035] Therefore, after the source package and the target package are split, the target file and the source file can be obtained whose data volume of a single file does not exceed the data volume threshold, so that the data volume of the subtask file group is smaller, thereby improving the efficiency of the working node in generating the first differential file according to the received subtask file group.

[0036] In a possible implementation, the above step 103 includes the following specific processing:

[0037] If among N target files, the arrangement order of the i-th target file in the target package is the same as the arrangement order of any source file in the source package, and the file name of the first file including the i-th target file in the target package is the same as the file name of the second file including any of the above source files in the source package, and the storage path of the first file in the target package is the same as the storage path of the second file in the source package, then the i-th target file and any of the above source files are formed into a subtask file group corresponding to the i-th target file; otherwise, the i-th target file is determined as the second differential file, wherein 1≤i≤N, the first file is the file in the target package including the i-th target file, and the second file is the file in the source package including any of the above source files.

[0038] Based on this, the above step 105 includes the following specific processing:

[0039] A differential package is generated based on at least part of the first differential file and at least part of the second differential file.

[0040] In a specific implementation, after each source file is obtained, the source file can be named so that the file name of the source file includes the arrangement order of the source file in the source package. For example, the source file can be named "H1_${c1}", H1 is the file name of the file in the source package that includes the source file (if the source file is obtained by splitting any file in the source package, then the file in the source package that includes the source file is the any file; if the source file is obtained by directly determining the file in the source package as the source file, then the file in the source package that includes the source file is the source file in the source package), c1 is the arrangement order of the source file in all the source files in the source package, for example, if any source file is the 5th source file in the source package, then c1=5; after each target file is obtained After the target file is generated, the target file can be named so that the file name of the target file includes the arrangement order of the target file in the target package. For example, the target file can be named "H2_${c2}", H2 is the file name of the file in the target package that includes the target file (if the target file is obtained by segmenting any file in the target package, then the file in the target package that includes the target file is the any file; if the target file is obtained by directly determining the file in the target package as the target file, then the file in the target package that includes the target file is the target file in the target package), c2 is the arrangement order of the target file in all the target files in the target package, for example, if any target file is the 7th target file in the target package, then c2=7, based on this:

[0041] For each target file, if there is a corresponding source file in the M source files for the target file, that is, the file name of the source file is the same as the file name of the target file (indicating that the arrangement order of the target files in the target package is the same as the arrangement order of the source files in the source package, and the file name of the file including the target file in the target package is the same as the file name of the file including the source file in the source package), and the storage path of the file including the target file in the target package is the same as the storage path of the file including the source file in the source package, then the target file and the source file form the subtask file group corresponding to the target file; otherwise, the target file is determined as the second differential file.

[0042] Optionally, after obtaining M source files and N target files and naming the M source files and N target files, the target management node can delete the source package and target package as well as the intermediate data generated during the segmentation of the target package and source package from the target management node to save space.

[0043] After determining all the first differential files and the second differential files, the target management node may package all the first differential files and all the second differential files into a zip compressed package, and use the compressed package as a differential package for users to download and use.

[0044] Therefore, the subtask file group is determined through the above processing, so that the source files and target files in the same subtask file group have the same arrangement order in the original package, the file names of the corresponding files in the original package are the same, and the storage paths in the original package are the same, so that the source files and target files in most subtask file groups are relatively similar, reducing the possibility that the target files with a low degree of similarity to all the source files are not directly determined as the second differential files, and then the target management node can directly determine more second differential files without going through the working node or the differential algorithm, further improving the efficiency of generating differential packages.

[0045] In a possible implementation, sending each subtask file group to a corresponding working node in step 104 includes the following specific processing:

[0046] Determine the maximum amount of memory corresponding to each subtask file group, wherein the maximum amount of memory corresponding to the subtask file group is the maximum amount of memory estimated to be used by any working node in the process of generating a corresponding first differential file based on the source file and the target file included in the subtask file group; and send each subtask file group in order from small to large to the working nodes whose free memory is greater than the corresponding maximum memory.

[0047] In a specific implementation, after determining each subtask file group, the target management node may match the subtask file group to a corresponding differential subtask, and adjust the status of the differential subtask to being executed. After determining each second differential file, the target management node may also match the second differential file to a corresponding differential subtask, and set the status of the differential subtask to success. Optionally, when all differential subtasks are matched, the target management node may write the information of these differential subtasks into a database for subsequent tracking and management. Based on this:

[0048] After determining each subtask file group, the target management node will also determine the maximum amount of memory corresponding to the subtask file group, and after determining the maximum amount of memory corresponding to each subtask file group, each subtask file group can be sent in order from small to large in the order of the maximum amount of memory corresponding to the subtask file group to the working node whose free memory is greater than the corresponding maximum amount of memory. In the process of sending each subtask file group to the working node, the target management node can try to obtain a globally unique distributed lock to ensure the atomicity and consistency of resource allocation.

[0049] Then, the work node resource table can be queried to find a work node that meets the condition that the amount of currently available memory (i.e., the amount of free memory) is greater than the maximum amount of memory corresponding to the subtask file group. If no work node that meets the condition is found, the target management node will release the lock and wait for (t+3) minutes before trying to obtain the lock and query again. If the required resources cannot be obtained after x cumulative attempts for the subtask file group, the target management node will set the status of the differential subtask corresponding to the subtask file group and all subsequent unexecuted differential subtasks to failure, and wait for the differential subtask corresponding to the previous subtask file to be executed before cleaning up the resources and notifying the user that the differential package production has failed. If a work node that meets the condition is found, the target management node will update the free memory of the work node in the database to the difference between the current free memory and the maximum memory corresponding to the subtask file group, and then release the distributed lock.

[0050] Then the target management node will try to establish a long connection with the working node based on the IP address of the working node. If the connection is successfully established, the management node will send the subtask file group to the working node and update the corresponding subtask record information in the database. If the connection fails to be established, the management node will send the IP address of the working node to all other management nodes to try to communicate. If all management nodes fail to communicate with the working node, the target management node believes that the working node has been offline by k8s and deletes the information of the working node in the working node resource table. If there is at least one management node that can successfully communicate with the working node, the target management node will update the free memory of the working node in the database to the sum of the current free memory and the maximum memory corresponding to the subtask file group, and the target management node will re-query the working node resource table to find a working node that meets the condition that the current available memory is greater than the maximum memory corresponding to the subtask file group.

[0051] Among them, t is the average time required for the working node to generate a corresponding first differential file based on the source file and the target file whose data volume is the data volume threshold using the differential algorithm. The unit of t is minutes. The differential algorithm can be a differential update algorithm (bsdiff algorithm), etc., and x can be set according to actual needs, which is not limited in the embodiments of the present application.

[0052] When the target management node queries the working node resource table to find a working node that meets the condition that the current available memory (i.e., free memory) is greater than the maximum memory corresponding to the subtask file group, if no working node that meets the condition is found, the target management node releases the lock and waits for (t+3) minutes to wait for k8s to realize automatic expansion and contraction functions, that is, when the memory usage of all working nodes is less than the target memory usage, a working node is added, and when the memory usage of all working nodes is greater than the target memory usage, a working node is reduced. The target memory usage is (b2-9.1K) / b2*100%, the minimum number of working nodes is 1 or 2, etc., the expansion cooling time is (t+1) min, and the contraction cooling time is (2t+2) min. As a result, computing resources can be automatically elastically scaled, thereby greatly improving the utilization of server resources.

[0053] In an embodiment of the present application, when each subtask file group is sent to a corresponding work node respectively, the maximum memory amount corresponding to each subtask file group can be determined, and each subtask file group can be sent in order from small to large in the corresponding maximum memory amount to a work node whose free memory amount is greater than the corresponding maximum memory amount. Thus, the subtask file group with a smaller maximum memory amount is first sent to the work node for processing, so that the generation of the first differential file can be completed faster, and the memory of the work node can be released as soon as possible for generating other first differential files, so as to reduce the peak number of work nodes used and save computing resources.

[0054] In a possible implementation, the formula for determining the maximum amount of memory corresponding to each subtask file group is as follows:

[0055] DV i =max(9n i +0.1,5n i +3m i +0.1);

[0056] Among them, DV i is the maximum amount of memory corresponding to the ith subtask file group among all subtask file groups, DV i The unit is gigabytes, n i is the data volume of the target file included in the i-th subtask file group, m i is the data volume of the source files included in the ith subtask file group, max() is the maximum value function, max(9n i +0.1,5n i +3m i +0.1) is 9n i +0.1 and 5n i +3m iMaximum value of +0.1.

[0057] Thus, the formula takes into account the memory requirements of the bsdiff algorithm when generating the suffix array, "9n i ” and “5n i +3m i " respectively reflects the memory usage of the algorithm when processing the target file and the source file, while "0.1" reserves additional memory space for other temporary variables. This formula can accurately determine the maximum amount of memory corresponding to each subtask file group, which can effectively avoid the problem of differential package generation failure due to insufficient memory resources.

[0058] In one possible implementation, the target management node is any management node among multiple management nodes, and the management nodes other than the target management node among the multiple management nodes are used to replace the target management node to work when the working state of the target management node is abnormal, that is, to update the target management node. Thereby, the possibility of failure of differential packet generation due to abnormal working state of the target management node can be reduced.

[0059] Optionally, after successfully establishing a long connection with the target management node and receiving the subtask file group, the working node will call the bsdiff algorithm to generate the first differential file. During the execution process, the working node will report the status and progress information of all differential subtasks it is currently executing to the corresponding target management node every 5 seconds for real-time monitoring and management. When the algorithm is executed, the working node will report the execution results to the corresponding management node to notify the task completion.

[0060] Optionally, during the process of the working node executing the differential sub-task, the target management node will continue to listen to the task reporting information from the working node to ensure that the task can proceed smoothly as expected. If the connection is abnormally disconnected or the sub-task information list reported by the working node is inconsistent with the record of the target management node during this process, the target management node will integrate all missing sub-tasks and re-query the working node to reallocate resources for these differential sub-tasks to ensure the integrity and reliability of the differential sub-tasks. After receiving the task execution result from the working node, the target management node will update the corresponding differential sub-task information in the database, and at the same time update the free memory amount of the working node in the database to the sum of the current free memory amount and the maximum memory amount corresponding to the sub-task file group, thereby completing the release of occupied node resources.

[0061] Optionally, when all differential subtasks have been executed by the corresponding working nodes, the target management node will perform a final status check. If it is confirmed that all first differential files and second differential files have been successfully generated, the management node will integrate these files into a zip compressed package as a differential package. If there is any differential subtask that fails to generate the first differential file, the target management node will promptly notify the user that the differential package generation has not been completed successfully. After all these operations are completed, the target management node will be responsible for cleaning up all source packages, target packages, and differential resource files that are no longer needed.

[0062] Optionally, all management nodes attempt to acquire a globally unique distributed lock at 0, 15, 30, and 45 minutes past the hour. The distributed lock can only be acquired by any one of the management nodes. The management node that successfully acquires the lock will be used as a check node to update the working node resource table to avoid working nodes being mistakenly deleted by the management node due to network problems. The specific process is as follows: the check node will obtain the IP address list of all currently online working nodes from the registration center, and compare it with the working node list recorded in the database. If nodes that have not been written to the database are found, the check node will try to communicate with these nodes to verify their status and information. If the communication is successful, the check node will write the IP addresses and memory information of these nodes into the data table for subsequent management and use.

[0063] Target management node

[0064] Corresponding to the above method embodiment, Figure 2 A schematic diagram of a target management node according to an embodiment of the present application is shown. Figure 2 As shown, the target management node 200 includes:

[0065] An acquisition unit 201 is used to acquire a source packet and a target packet for generating a differential packet;

[0066] A splitting unit 202, used to split the source package into multiple source files, and split the target package into multiple target files;

[0067] A determination unit 203 is used to determine a corresponding subtask file group according to at least part of the target file, wherein the subtask file group corresponding to the target file includes the target file and a corresponding source file;

[0068] The sending unit 204 is used to send each subtask file group to the corresponding working node, so that each working node generates a corresponding first differential file according to the source file and the target file included in the received subtask file group, and sends the first differential file to the target management node, wherein each working node is connected to the target management node;

[0069] The generating unit 205 is configured to generate a differential package according to at least part of the first differential file.

[0070] It should be noted that the target management node of this embodiment is used to implement the corresponding differential packet generation method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0071] Electronic devices

[0072] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Figure 3 As shown, the electronic device may include: a processor 302, a communications interface 304, a memory 306, and a communications bus 308. Among them:

[0073] The processor 302 , the communication interface 304 , and the memory 306 communicate with each other via the communication bus 308 .

[0074] The communication interface 304 is used to communicate with other electronic devices or servers.

[0075] The processor 302 is used to execute the program 310, and specifically can execute the relevant steps in any of the above-mentioned differential packet generation method embodiments.

[0076] Specifically, the program 310 may include program codes, which include computer operation instructions.

[0077] The processor 302 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0078] RISC-V is an open source instruction set architecture based on the Reduced Instruction Set (RISC) principle. It can be applied to various aspects such as single-chip microcomputers and FPGA chips. Specifically, it can be applied to the fields of IoT security, industrial control, mobile phones, personal computers, etc., and because it is designed with small, fast, and low-power realities in mind, it is particularly suitable for modern computing devices such as warehouse-scale cloud computers, high-end mobile phones, and tiny embedded systems. With the rise of artificial intelligence Internet of Things (AIoT), the RISC-V instruction set architecture has also received more and more attention and support, and is expected to become the next generation of widely used CPU architecture.

[0079] The computer operation instructions in the embodiment of the present application may be computer operation instructions based on the RISC-V instruction set architecture, and correspondingly, the processor 302 may be designed based on the RISC-V instruction set. Specifically, the chip of the processor in the electronic device provided in the embodiment of the present application may be a chip designed with the RISC-V instruction set, and the chip may execute executable code based on the configured instructions, thereby implementing the differential packet generation method in the above embodiment.

[0080] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0081] The program 310 may be specifically used to enable the processor 302 to execute the differential packet generation method in any of the aforementioned embodiments.

[0082] The specific implementation of each step in program 310 can refer to the corresponding description of the corresponding steps and units in any of the above-mentioned differential packet generation method embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0083] It should be noted that this embodiment has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0084] Computer storage media

[0085] The present application also provides a computer-readable storage medium storing instructions for causing a machine to execute the differential packet generation method as described herein. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0086] In this case, the program code read from the storage medium itself can implement the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.

[0087] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0088] Computer program product

[0089] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0090] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0091] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0092] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0093] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0094] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0095] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A differential packet generation method, characterized in that: Applied to a target management node, the method comprises: Obtain source and target packages for generating differential packages; Splitting the source package into multiple source files, and splitting the target package into multiple target files; Determine a corresponding subtask file group according to at least part of the target file, wherein the subtask file group corresponding to the target file includes the target file and a corresponding one of the source files; Sending each of the subtask file groups to a corresponding work node respectively, so that each of the work nodes generates a corresponding first differential file according to the source file and the target file included in the received subtask file group, and sends the first differential file to the target management node, wherein each of the work nodes is connected to the target management node; A differential package is generated based on at least part of the first differential file.

2. The method according to claim 1, characterized in that The step of dividing the source package into a plurality of source files and dividing the target package into a plurality of target files comprises: Divide the source package into M source files, wherein the data volume of each source file is less than or equal to a data volume threshold, M≥2; The target package is divided into N target files, wherein the data volume of each target file is less than or equal to the data volume threshold, and N≥2.

3. The method according to claim 2, characterized in that The determining of the corresponding subtask file group according to at least part of the target file includes: If, among the N target files, the arrangement order of the i-th target file in the target package is the same as the arrangement order of any source file in the source package, and the file name of the first file in the target package including the i-th target file is the same as the file name of the second file in the source package including any source file, and the storage path of the first file in the target package is the same as the storage path of the second file in the source package, then the i-th target file and the any source file are formed into a subtask file group corresponding to the i-th target file, otherwise, the i-th target file is determined as the second differential file, wherein 1≤i≤N; Generating a differential package according to at least part of the first differential file includes: The differential package is generated according to at least part of the first differential file and at least part of the second differential file.

4. The method according to claim 1, characterized in that: The sending each of the subtask file groups to the corresponding working node respectively includes: Determine the maximum amount of memory corresponding to each of the subtask file groups, wherein the maximum amount of memory corresponding to the subtask file group is the maximum amount of memory used by the working node memory when any of the working nodes generates the corresponding first differential file according to the source file and the target file included in the subtask file group; In order from small to large corresponding maximum memory amounts, each of the subtask file groups is sent in sequence to the working node whose free memory amount is greater than the corresponding maximum memory amount.

5. The method according to claim 4, characterized in that The formula for determining the maximum amount of memory corresponding to each subtask file group is as follows: <h2 style=";text-align:left;direction:ltr">DV<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =max(9n<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +0.1, 5n<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +3m<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +0.1); Among them, DV i is the maximum memory size corresponding to the ith subtask file group among all the subtask file groups, DV i The unit is gigabytes, n i is the data volume of the target file included in the i-th subtask file group, m i is the data volume of the source files included in the i-th subtask file group, max() is the maximum value function, max(9n i +0.1,5n i +3m i +0.1) is 9n i +0.1 and 5n i +3m i Maximum value of +0.

1.

6. The method according to claim 1, characterized in that The target management node is any one of the multiple management nodes, and the management nodes other than the target management node among the multiple management nodes are used to replace the target management node to work when the working state of the target management node is abnormal.

7. A target management node, characterized in that: The target management node includes: An acquisition unit, used for acquiring a source packet and a target packet for generating a differential packet; A splitting unit, used for splitting the source package into a plurality of source files, and splitting the target package into a plurality of target files; a determining unit, configured to determine a corresponding subtask file group according to at least part of the target file, wherein the subtask file group corresponding to the target file includes the target file and a corresponding one of the source files; A sending unit, used to send each of the subtask file groups to a corresponding working node, so that each of the working nodes generates a corresponding first differential file according to the source file and the target file included in the received subtask file group, and sends the first differential file to the target management node, wherein each of the working nodes is connected to the target management node; A generating unit is used to generate a differential package according to at least part of the first differential file.

8. An electronic device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the method as claimed in any one of claims 1 to 6.

9. A computer storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer instructions for instructing a computing device to execute the method as claimed in any one of claims 1 to 6.

Images