Data processing method and device, computer equipment and storage medium

By determining the number of concurrency and backup methods based on the performance of the host in Cinder, and performing multi-threaded backup processing on the data, the problems of slow backup speed and low performance utilization of S3 object storage are solved, and more efficient data processing and network performance utilization are achieved .

CN119003239APending Publication Date: 2024-11-22ZHENGZHOU INSPUR DATA TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411011256.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When Cinder uses S3 object storage backup, it is linear operation, which causes the data disk size to be too large and the processing time is too long. The existing method can only utilize 10% of the performance of the data network, which has huge performance. loss.

Method used

When detecting data backup requests, the target concurrency number is determined based on the host memory and network card performance, the data is cut into pieces, and the full or incremental backup method is selected according to the number of backups, and multi-threaded backup is used to make full use of computing power and Transmission capability.

Benefits of technology

It shortens the backup time, fully utilizes the performance of physical machines, improves network performance utilization and data processing efficiency, and avoids the problem of high vacancy rate of physical machines.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003239A_ABST
    Figure CN119003239A_ABST
Patent Text Reader

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium, and belongs to the technical field of cloud computing. The method comprises the steps that when a data backup request is detected in response, data needing to be backed up is acquired; determining a target concurrency number based on the host machine memory and the host machine network card performance; cutting data needing to be backed up according to the target concurrent number to obtain at least one piece of fragment block data; based on the backup times of the target fragment block data, determining a backup mode of the target fragment block data, the backup mode being full backup or incremental backup; and according to the backup mode and the target concurrent number, performing multi-thread transmission backup on the data of the plurality of fragmentation blocks. The backup time can be shortened, the physical machine performance can be fully and effectively utilized, the problem that the performance vacancy rate of the physical machine is high is avoided, and the network performance utilization rate and the data processing efficiency are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] When Cinder (a component that provides block storage services in the OpenStack-open source cloud computing management platform project) uses S3 (Simple Storage Service) object storage for backup, it is a linear operation that reads hard disk data in fixed-size data blocks in sequence and uploads the data to the S3 object storage in sequence. The current mode is simple and too linear. This method is often too slow and takes too long to process when the size of the data disk is too large. In addition, the existing processing method can only utilize 10% of the performance of the data network, resulting in huge performance loss.

[0003] Therefore, there is an urgent need to propose a data processing method, device, computer equipment and storage medium that can improve network performance utilization and data processing efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment and storage medium that can improve network performance utilization and data processing efficiency in response to the above technical problems.

[0005] In a first aspect, a data processing method is provided, the method comprising:

[0006] In response to detecting a data backup request, obtaining data that needs to be backed up;

[0007] Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0008] The data to be backed up is split according to the target concurrent number to obtain at least one shard block data;

[0009] Based on the number of backups of the target shard block data, determine a backup mode for the target shard block data, the backup mode being full backup or incremental backup;

[0010] According to the backup mode and the target concurrency number, multiple shard block data are backed up through multi-threaded transmission.

[0011] Optionally, based on the host memory and host network card performance, the target concurrency number is determined including:

[0012] Based on the host machine memory, obtaining reserved memory;

[0013] Based on the host network card performance, obtain the reserved bandwidth;

[0014] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0015]

[0016] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0017]

[0018] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0019] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0020] Based on the comparison result, a target concurrency number is determined.

[0021] Optionally, based on the comparison result of the concurrent number of the computing capability and the concurrent number of the transmission capability, determining the target concurrent number includes:

[0022] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0023] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0024] Optionally, determining the transmission backup time based on the target concurrent number includes:

[0025] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0026]

[0027] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0028] Optionally, based on the number of backups of the target shard block data, determining a backup mode of the target shard block data, the backup mode being full backup or incremental backup including:

[0029] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup;

[0030] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0031] Optionally, in response to detecting that the backup mode of the target shard block data is full backup, the method includes:

[0032] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0033] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set;

[0034] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data;

[0035] The compressed target shard block data is transferred and backed up.

[0036] Optionally, in response to detecting that the backup mode of the target shard block data is incremental backup, the method includes:

[0037] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0038] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set;

[0039] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data;

[0040] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data;

[0041] The compressed target shard block data is transferred and backed up.

[0042] In a second aspect, a data processing device is provided, the device comprising:

[0043] A data acquisition module is used to acquire data that needs to be backed up when a data backup request is detected;

[0044] A concurrent number determination module, used to determine the concurrent number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0045] A shard block data acquisition module, used to split the data to be backed up according to the concurrency number to obtain at least one shard block data;

[0046] A backup mode determination module, used to determine the backup mode of the target shard block data based on the number of backups of the target shard block data, wherein the backup mode is full backup or incremental backup;

[0047] The transmission backup module is used to perform multi-threaded transmission backup of multiple shard block data according to the backup mode and the concurrency number.

[0048] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0049] In response to detecting a data backup request, obtaining data that needs to be backed up;

[0050] Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0051] The data to be backed up is split according to the target concurrent number to obtain at least one shard block data;

[0052] Based on the number of backups of the target shard block data, determine a backup mode for the target shard block data, the backup mode being full backup or incremental backup;

[0053] According to the backup mode and the target concurrency number, multiple shard block data are backed up through multi-threaded transmission.

[0054] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0055] In response to detecting a data backup request, obtaining data that needs to be backed up;

[0056] Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0057] The data to be backed up is split according to the target concurrent number to obtain at least one shard block data;

[0058] Based on the number of backups of the target shard block data, determine a backup mode for the target shard block data, the backup mode being full backup or incremental backup;

[0059] According to the backup mode and the target concurrency number, multiple shard block data are backed up through multi-threaded transmission.

[0060] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0061] In response to detecting a data backup request, obtaining data that needs to be backed up;

[0062] Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0063] The data to be backed up is split according to the target concurrent number to obtain at least one shard block data;

[0064] Based on the number of backups of the target shard block data, determine a backup mode for the target shard block data, the backup mode being full backup or incremental backup;

[0065] According to the backup mode and the target concurrency number, multiple shard block data are backed up through multi-threaded transmission.

[0066] The above-mentioned data processing method, device, computer equipment and storage medium, the method includes: in response to detecting a data backup request, obtaining data that needs to be backed up; determining the target concurrency based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data that needs to be backed up is located; cutting the data that needs to be backed up according to the target concurrency to obtain at least one shard block data; determining the backup method of the target shard block data based on the number of backups of the target shard block data, the backup method being full backup or incremental backup; performing multi-threaded transmission backup on multiple shard block data according to the backup method and the target concurrency. The present application performs multi-threaded backup processing on data based on the concurrency, which can shorten the backup time, fully and effectively utilize the performance of the physical machine, avoid the problem of high performance idle rate on the physical machine, and improve the network performance utilization and data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0068] Figure 2 is a flow chart of a data processing method in one embodiment;

[0069] Figure 3 A schematic diagram of a single-threaded data backup process of a data processing method in one embodiment;

[0070] Figure 4 A schematic diagram of a multi-threaded data backup process of a data processing method in one embodiment;

[0071] Figure 5 A schematic diagram of a concurrent number acquisition process of a data processing method in one embodiment;

[0072] Figure 6 is a structural block diagram of a data processing device in one embodiment;

[0073] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0076] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0077] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0078] According to the background technology, when Cinder uses S3 object storage for backup, it is a linear operation, reading hard disk data in fixed-size data blocks in sequence, and uploading the data to S3 object storage in sequence. This method is often too slow when the size of the data disk is too large.

[0079] In order to solve the above technical problems, the present application provides a data processing method, device, equipment and storage medium, which shortens the backup time by optimizing the data processing time, fully and effectively utilizes the performance of the physical machine, avoids the problem of high performance idle rate on the physical machine, and improves the network performance utilization and data processing efficiency.

[0080] The data processing method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the data processing platform set on the server 104 through the network, wherein the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0081] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0082] S1: In response to detecting a data backup request, obtaining data that needs to be backed up.

[0083] It should be noted that data backup is the basis of disaster recovery. It refers to the process of copying all or part of the data set from the hard disk or array of the application host to other storage media to prevent data loss due to system operation errors or system failures. The data that needs to be backed up generally refers to the data that needs to be backed up in the host, such as personal, family and work documents, pictures, audio, video and other personal files, mailboxes and address book data, etc. It generally uses the OpenStack block storage service (Cinder), which adds persistent storage to virtual machines. Block storage provides an infrastructure for managing volumes and interacts with OpenStack computing services to provide volumes for instances. This service also activates the functions of managing volume snapshots and volume types. OpenStack is a cloud platform management project. It is an open source project composed of several main components to provide software for the construction and management of public clouds, private clouds and hybrid clouds. Cinder Backup is a functional module of the OpenStack block storage service, which is used to back up and restore block storage volume data. In OpenStack, Cinder is used to provide persistent block storage services, and Cinder Backup provides backup and restore functions for these volumes. Using Cinder Backup can create snapshots of volumes and back them up to specified backup targets, such as object storage (such as Swift) or other supported back-end storage. Backups can be migrated between different storage backends to achieve flexibility and redundancy. At the same time, Cinder Backup also supports incremental backups. Incremental backups can be created regularly as needed to reduce the size and transmission time of backup data. The backed-up data can be restored as needed. You can choose to restore the entire volume or restore a specific data version from the backup. Cinder Backup also provides some management and monitoring functions for managing backup tasks, checking backup status, and executing backup strategies; this application is based on Cinder using S3 object storage backup. S3 (Simple Storage Service) storage is a highly scalable object storage service for storing and retrieving data of any type and size. S3 supports unlimited storage capacity and can easily store and process large-scale data. It can automatically expand to adapt to growing storage needs and supports high concurrent access to meet large-scale data read and write operations.

[0084] S2: Determine the target concurrency based on the host memory and host network card performance.

[0085] It should be noted that the host machine is the host where the data to be backed up is located. The host machine memory mainly refers to a part of the computing resources provided to users in a cloud computing environment, including CPU and memory. The network connection speed and quality that the host machine network card can achieve, and its performance is generally reflected in bandwidth.

[0086] In some specific embodiments, Figure 5 As shown in the figure, based on the host memory and the host network card performance, the target concurrency number is determined as follows:

[0087] Based on the host machine memory, the reserved memory is obtained, wherein the system tool free can be used to obtain the physical machine memory value, that is, the host machine memory, and the reserved memory can be obtained by reading the reserved memory percentage in the configuration file. Generally, the reserved memory can be set according to actual needs, such as 20%. If there is no special statement in the configuration file, 20% of the memory is reserved by default. If the value is lower than 20% or higher than 20%, it will put pressure on the physical machine and cause the host machine performance to deteriorate;

[0088] Based on the host network card performance, the reserved bandwidth is obtained, wherein the system tool ethtool is used to obtain the network card bandwidth, which is used to describe the host network card performance. The reserved bandwidth can be obtained by reading the reserved memory percentage in the configuration file. Generally, the reserved bandwidth can be set according to actual needs, such as 20%. If there is no special statement in the configuration file, 20% of the memory is reserved by default. If the value is lower than 20% or higher than 20%, it will put pressure on the physical machine and cause the host performance to deteriorate;

[0089] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0090]

[0091] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0092]

[0093] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0094] The values ​​of the above-mentioned concurrent computing capability and concurrent transmission capability are all subjected to modulo operation to obtain the final concurrent computing capability and concurrent transmission capability.

[0095] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0096] Based on the comparison result, a target concurrency number is determined.

[0097] In some specific implementations, based on the comparison result of the concurrent number of computing capability and the concurrent number of transmission capability, determining the target concurrent number includes:

[0098] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0099] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0100] Specifically, before performing data backup, you need to obtain a reasonable concurrency number. The concurrency number changes with the physical performance of the host machine and does not need to change at any time, so it is generally initialized when the service starts. After obtaining the concurrency number of computing power and the concurrency number of transmission capacity, compare the two. If the selected concurrency number is greater than the minimum concurrency number, it will put too much pressure on other capabilities. When there is pressure, it will cause the time spent to be too long. Therefore, take the minimum value as the optimal concurrency number.

[0101] In the above implementation, based on influencing factors such as computing power and transmission capacity, a reasonable number of concurrency is calculated and determined, which can shorten the backup time exponentially, thereby enabling efficient processing of backup files without increasing system pressure.

[0102] S3: Split the data to be backed up according to the target concurrency number to obtain at least one shard block of data.

[0103] It should be noted that cutting the data to be backed up refers to dividing the backup file into multiple pieces of data.

[0104] Specifically, when data backup is needed, first read the global variable concurrency number, initialize an array of the concurrency number, and switch the backup file according to the concurrency number. For example, if the concurrency number is 2, the backup file is split into two files, that is, the data to be backed up is cut into two shards.

[0105] In the above implementation, the data to be backed up is segmented based on the number of concurrencies, and the corresponding backup files are assigned to different threads to process, thereby realizing multi-threaded data processing and improving data processing efficiency.

[0106] S4: Based on the number of backup times of the target shard block data, determine a backup mode of the target shard block data, where the backup mode is full backup or incremental backup.

[0107] It should be noted that the number of backups of the target shard block data refers to whether the shard block data has been backed up before this backup and the number of times it has been backed up. Full backup refers to the method of backing up the shard block data when it has not been backed up. Incremental backup refers to the method of backing up the shard block data when it has been backed up. Full backup and incremental backup are two common backup strategies. Full backup refers to backing up all data in the database, while incremental backup only backs up data that has changed since the last full backup or the most recent incremental backup.

[0108] In some specific implementations, based on the number of backup times of the target shard block data, determining the backup method of the target shard block data includes:

[0109] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is full backup, wherein the first preset threshold can be set according to actual needs. The first preset threshold of the present application is preferably 0, that is, when the number of backups is 0, determining that the backup mode of the target shard block data is full backup;

[0110] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0111] In the above implementation, by selecting different backup methods, it is ensured that the same data will not be backed up repeatedly, which wastes time and resources and reduces the performance loss of the physical machine.

[0112] S5: Perform multi-threaded transmission backup on multiple shard block data according to the backup mode and the target concurrency number.

[0113] It should be noted that the number of threads is the same as the target concurrency number. The threads are initialized according to the concurrency array. Each thread only processes the current split backup file, such as Figure 3 As shown, the processing flow of each thread is as follows: read the fixed-size backup shard block data; determine whether it is a full backup; calculate the sha256 of the incremental block corresponding to the current shard block data; if it is a full backup, the current shard block data needs to be recorded. If it is not a full backup, it is necessary to compare the previous backup, compare its sha256 and data offset. If there is a change, the current shard block data needs to be recorded. If there is no change, the current shard block data is not recorded; compress the current shard block data; transmit the compressed current shard block data for backup, such as Figure 4 As shown, it is the data backup processing flow under multiple threads.

[0114] Specifically, in response to detecting that the backup mode of the target shard block data is full backup, the method includes:

[0115] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks, wherein the preset incremental block size is a method of dividing a string into smaller parts, which can be set according to actual needs, such as an incremental block size of 64MB;

[0116] Based on a cryptographic hash function, the plurality of incremental blocks are calculated to generate a first sha256 set, wherein the cryptographic hash function is a Hash algorithm, and sha256 is a hash value, that is, the input information of any length is converted into a hash value of a fixed length (usually 160 bits) by the Hash algorithm, thereby generating the first sha256 set;

[0117] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data, wherein the data offset is the location information of the data in the file. When a full backup is performed, the data offset and sha256 set corresponding to the target shard block data need to be recorded and compressed for backup;

[0118] The compressed target shard block data is transferred and backed up.

[0119] Furthermore, in response to detecting that the backup mode of the target shard block data is incremental backup, the method includes:

[0120] Based on the preset incremental block size, the target shard block data is divided to obtain multiple incremental blocks. Similarly, the preset incremental block size is a method of dividing a string into smaller parts, which can be set according to actual needs, such as the incremental block size is 64MB, etc.;

[0121] Based on a cryptographic hash function, the plurality of incremental blocks are calculated to generate a second sha256 set. Similarly, the cryptographic hash function is a Hash algorithm, that is, the input information of any length is converted into a hash value of a fixed length (usually 160 bits) by the Hash algorithm, thereby generating the second sha256 set.

[0122] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data, wherein the parent backup shard block data refers to the shard block data of the last full backup or incremental backup, which can be used as the parent of other increments, and has the third sha256 set and the second data offset, which are stored in the database;

[0123] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, the third data offset of the target shard block data and the second sha256 set are recorded, and the target shard block data is compressed, wherein as long as the current sha256 set is different from the sha256 set corresponding to the previous backup, and / or the data offsets of the two are different, the current shard block data and its corresponding sha256 set and data offset are compressed and backed up, and if the sha256 sets and data offsets of the two are the same, no processing is performed and the backup is terminated directly;

[0124] The compressed target shard block data is transferred and backed up.

[0125] In some specific implementations, determining the transmission backup time based on the target concurrent number includes:

[0126] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0127]

[0128] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0129] Data processing time: Data processing is related to the performance of the physical machine. Generally speaking, 100MB of data is insignificant for a physical machine, but the data block processing capability needs to be considered. Data processing operations include calculating sha256 and compressing data. If the data block is too large, the time for these two data operations will increase. This increase is often increased in an idempotent way. Through actual measurements of 100MB of data, these two times are the most appropriate. At the same time, in the s3 protocol, 100MB means that in actual transmission, the protocol often does not need to cut data, but 100MB processing is very time-consuming for a physical machine. It is said that there is a huge performance loss. If 100MB of data can be processed concurrently, the speed can be greatly improved. For example, if 10 100MB data are processed simultaneously, the time spent can be reduced tenfold. For example, there is a 500G data disk. To process each block of data as a 100MB data block, a single thread must traverse the 500G data block in sequence and process the data. The processing time is about 4.2 hours, which is too long. However, after giving a reasonable number of concurrent processes, the transmission backup time can be greatly shortened by processing the same backup file through multiple threads. Data upload time: In a production environment, the data network transmission bandwidth is often a 10G data network, and it can transmit 1000MB of data per second at the same time. The higher the capacity of the data network, the shorter the upload time. The existing processing method can only utilize 10% of the performance of the data network, which has a huge performance loss. If it can be uploaded concurrently, the network performance can be greatly utilized. The larger the number of concurrency, the shorter the total time. However, the larger the number of concurrency, the better. If the number of concurrency is too large, it will cause excessive pressure on the physical machine, resulting in performance degradation. In order to obtain a reasonable and efficient number of concurrency, it is necessary to consider the impact of the number of concurrency. Influencing factors: For time analysis, it can be seen that processing time is related to computing power, upload time is related to transmission capacity, computing power is affected by physical CPU memory, compression and verification are the most time-consuming calculations, and the existing calculation method only has a few percent utilization of physical CPU memory. To increase the utilization of physical CPU memory, concurrent calculation is required. Transmission capacity is affected by the performance of the physical network card. In a formal environment, the physical network card is often only one-tenth utilized, and concurrent processing is also required. Based on this, a reasonable number of concurrency is determined through calculation, so that backup files can be processed efficiently without increasing system pressure.

[0130] In the above implementation, by selecting the backup method and target concurrency, multiple shard block data are transmitted and backed up in a multi-threaded manner, which can shorten the backup time exponentially, optimize the user experience, reduce the user's backup waiting time, make full use of the performance of the physical machine, and reduce the performance loss of the physical machine.

[0131] In some specific implementations, after transmitting the backup target shard block data, the method further includes:

[0132] Obtain a first target hash value and a first target data offset of the target shard block data before transmission, and a second target hash value and a second target data offset of the target shard block data after transmission, wherein the hash value is the value in the sha256 set obtained by calculating the cryptographic hash function;

[0133] Compare the first target hash value and the second target hash value, the first target data offset and the second target data offset respectively, and obtain a comparison result;

[0134] In response to detecting that the first target hash value and the second target hash value are different, and / or the first target data offset and the second target data offset are different, obtaining the corresponding deviation, wherein the deviation refers to the value of the difference point when the two are different;

[0135] In response to detecting that the deviation is greater than a second preset threshold, a prompt message is sent to the user terminal to remind the user to check, wherein the second preset threshold can be set according to actual needs, such as the number of deviations is 3;

[0136] In response to detecting that the deviation amount is less than or equal to the second preset threshold and greater than or equal to the third preset threshold, marking the thread performing the backup transmission, and if the number of markings is greater than the fourth preset threshold, sending a prompt message to the user terminal to remind the user to check, wherein the third preset threshold can be set according to actual needs, such as the number of deviations is 1, etc., and the fourth preset threshold can also be set according to actual needs, such as the number of markings is 2;

[0137] In response to detecting that the first target hash value and the second target hash value are the same, and the first target data offset and the second target data offset are the same, no processing is required.

[0138] In the above implementation, the target shard block data after transmission backup is verified, thereby improving the reliability of transmission backup and enhancing user experience.

[0139] In the above-mentioned data processing method, the method includes: in response to detecting a data backup request, obtaining data that needs to be backed up; determining the target concurrency based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data that needs to be backed up is located; cutting the data that needs to be backed up according to the target concurrency to obtain at least one shard block data; determining the backup method of the target shard block data based on the number of backups of the target shard block data, wherein the backup method is full backup or incremental backup; performing multi-threaded transmission backup on multiple shard block data according to the backup method and the target concurrency. The present application performs multi-threaded backup processing on data based on the concurrency, which can shorten the backup time, fully and effectively utilize the performance of the physical machine, avoid the problem of high performance idle rate on the physical machine, and improve the network performance utilization and data processing efficiency.

[0140] It should be understood that although Figure 2-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0141] In one embodiment, Figure 6 As shown, a data processing device is provided, including: a data acquisition module, a concurrent number determination module, a shard block data acquisition module, a backup mode determination module and a transmission backup module, wherein:

[0142] A data acquisition module is used to acquire data that needs to be backed up when a data backup request is detected;

[0143] A concurrent number determination module, used to determine a target concurrent number based on a host machine memory and a host machine network card performance, wherein the host machine is a host machine where the data to be backed up is located;

[0144] A shard block data acquisition module, used to split the data to be backed up according to the target concurrency number to obtain at least one shard block data;

[0145] A backup mode determination module, used to determine the backup mode of the target shard block data based on the number of backups of the target shard block data, wherein the backup mode is full backup or incremental backup;

[0146] The transmission backup module is used to perform multi-threaded transmission backup of multiple shard block data according to the backup mode and the target concurrency number.

[0147] As a preferred implementation, in the embodiment of the present invention, the concurrent number determination module is specifically used for:

[0148] Based on the host machine memory, obtaining reserved memory;

[0149] Based on the host network card performance, obtain the reserved bandwidth;

[0150] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0151]

[0152] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0153]

[0154] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0155] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0156] Based on the comparison result, a target concurrency number is determined.

[0157] As a preferred implementation, in the embodiment of the present invention, the concurrent number determination module is further used to:

[0158] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0159] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0160] As a preferred implementation, in the embodiment of the present invention, the device further includes a transmission backup time determination module, and the transmission backup time determination module is specifically used to:

[0161] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0162]

[0163] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0164] As a preferred implementation, in the embodiment of the present invention, the backup mode determination module is specifically used for:

[0165] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup;

[0166] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0167] As a preferred implementation, in the embodiment of the present invention, the backup mode determination module is further used to:

[0168] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0169] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set;

[0170] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data;

[0171] The compressed target shard block data is transferred and backed up.

[0172] As a preferred implementation, in the embodiment of the present invention, the backup mode determination module is further used to:

[0173] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0174] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set;

[0175] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data;

[0176] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data;

[0177] The compressed target shard block data is transferred and backed up.

[0178] For the specific definition of the data processing device, please refer to the definition of the data processing method above, which will not be repeated here. Each module in the above data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0179] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0180] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0182] S1: in response to detecting a data backup request, obtaining data that needs to be backed up;

[0183] S2: Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0184] S3: Splitting the data to be backed up according to the target concurrent number to obtain at least one shard block of data;

[0185] S4: Based on the number of backup times of the target shard block data, determine the backup mode of the target shard block data, where the backup mode is full backup or incremental backup;

[0186] S5: Perform multi-threaded transmission backup on multiple shard block data according to the backup mode and the target concurrency number.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] Based on the host machine memory, obtaining reserved memory;

[0189] Based on the host network card performance, obtain the reserved bandwidth;

[0190] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0191]

[0192] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0193]

[0194] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0195] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0196] Based on the comparison result, a target concurrency number is determined.

[0197] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0198] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0199] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0200] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0201] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0202]

[0203] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0204] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0205] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup;

[0206] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0207] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0208] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0209] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set;

[0210] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data;

[0211] The compressed target shard block data is transferred and backed up.

[0212] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0213] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0214] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set;

[0215] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data;

[0216] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data;

[0217] The compressed target shard block data is transferred and backed up.

[0218] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0219] S1: in response to detecting a data backup request, obtaining data that needs to be backed up;

[0220] S2: Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0221] S3: Splitting the data to be backed up according to the target concurrent number to obtain at least one shard block of data;

[0222] S4: Based on the number of backup times of the target shard block data, determine the backup mode of the target shard block data, where the backup mode is full backup or incremental backup;

[0223] S5: Perform multi-threaded transmission backup on multiple shard block data according to the backup mode and the target concurrency number.

[0224] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0225] Based on the host machine memory, obtaining reserved memory;

[0226] Based on the host network card performance, obtain the reserved bandwidth;

[0227] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0228]

[0229] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0230]

[0231] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0232] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0233] Based on the comparison result, a target concurrency number is determined.

[0234] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0235] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0236] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0237] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0238] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0239]

[0240] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0241] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0242] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup;

[0243] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0244] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0245] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0246] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set;

[0247] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data;

[0248] The compressed target shard block data is transferred and backed up.

[0249] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0250] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0251] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set;

[0252] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data;

[0253] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data;

[0254] The compressed target shard block data is transferred and backed up.

[0255] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program when executed by a processor implements the following steps:

[0256] S1: in response to detecting a data backup request, obtaining data that needs to be backed up;

[0257] S2: Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located;

[0258] S3: Splitting the data to be backed up according to the target concurrent number to obtain at least one shard block of data;

[0259] S4: Based on the number of backup times of the target shard block data, determine the backup mode of the target shard block data, where the backup mode is full backup or incremental backup;

[0260] S5: Perform multi-threaded transmission backup on multiple shard block data according to the backup mode and the target concurrency number.

[0261] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0262] Based on the host machine memory, obtaining reserved memory;

[0263] Based on the host network card performance, obtain the reserved bandwidth;

[0264] Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes:

[0265]

[0266] Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes:

[0267]

[0268] Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth;

[0269] Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result;

[0270] Based on the comparison result, a target concurrency number is determined.

[0271] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0272] In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency;

[0273] In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

[0274] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0275] The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes:

[0276]

[0277] Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

[0278] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0279] In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup;

[0280] In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

[0281] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0282] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0283] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set;

[0284] Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data;

[0285] The compressed target shard block data is transferred and backed up.

[0286] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0287] Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks;

[0288] Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set;

[0289] Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data;

[0290] In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data;

[0291] The compressed target shard block data is transferred and backed up.

[0292] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0293] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0294] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A data processing method, It is characterized in that The method comprises: In response to detecting a data backup request, obtaining data that needs to be backed up; Determine the target concurrency number based on the host machine memory and the host machine network card performance, wherein the host machine is the host where the data to be backed up is located; The data to be backed up is split according to the target concurrent number to obtain at least one shard block data; Based on the number of backups of the target shard block data, determine a backup mode for the target shard block data, the backup mode being full backup or incremental backup; According to the backup mode and the target concurrency number, multiple shard block data are backed up through multi-threaded transmission.

2. The data processing method according to claim 1, It is characterized in that Based on the host memory and host network card performance, determine the target concurrency number including: Based on the host machine memory, obtaining reserved memory; Based on the host network card performance, obtain the reserved bandwidth; Based on the reserved memory and a first preset function, the concurrent number of computing capabilities is calculated and determined, wherein the first preset function includes: Based on the reserved bandwidth and the second preset function, the number of concurrent transmission capabilities is calculated and determined, wherein the second preset function includes: Among them, J represents the number of concurrent computing capabilities, C represents the number of concurrent transmission capabilities, and w 1 Indicates physical memory, b 1 Indicates the percentage of reserved memory, s indicates the size of the data block, and b 2 Indicates the percentage of reserved bandwidth, w 2 Indicates physical bandwidth; Comparing the concurrent number of computing capability and the concurrent number of transmission capability to obtain a comparison result; Based on the comparison result, a target concurrency number is determined.

3. The data processing method according to claim 2, It is characterized in that Based on the comparison result of the concurrent number of the computing capability and the concurrent number of the transmission capability, determining the target concurrent number includes: In response to detecting that the computing capability concurrency is less than the transmission capability concurrency, defining the computing capability concurrency as a target concurrency; In response to detecting that the computing capability concurrency is greater than the transmission capability concurrency, defining the transmission capability concurrency as the target concurrency.

4. The data processing method according to any one of claims 1 to 3, It is characterized in that Based on the target number of concurrent connections, determining the transmission backup time includes: The transmission backup time is calculated and determined based on a third preset function and the target concurrency number, wherein the third preset function includes: Where k represents the target concurrent number, T represents the transmission backup time, r represents the hard disk data size, a represents the data processing time, c represents the data transmission time, and s represents the size of the data block.

5. The data processing method according to claim 1, It is characterized in that Based on the number of backups of the target shard block data, determining a backup mode of the target shard block data, wherein the backup mode is a full backup or an incremental backup and includes: In response to detecting that the number of backups of the target shard block data is less than or equal to a first preset threshold, determining that the backup mode of the target shard block data is a full backup; In response to detecting that the number of backups of the target shard block data is greater than a first preset threshold, determining that the backup mode of the target shard block data is incremental backup.

6. The data processing method according to claim 5, It is characterized in that In response to detecting that the backup mode of the target shard block data is full backup, the method includes: Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks; Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a first sha256 set; Record the first data offset of the target shard block data and the first sha256 set, and compress the target shard block data; The compressed target shard block data is transferred and backed up.

7. The data processing method according to claim 5, It is characterized in that In response to detecting that the backup mode of the target shard block data is incremental backup, the method includes: Based on a preset incremental block size, the target shard block data is divided to obtain a plurality of incremental blocks; Based on a cryptographic hash function, calculating the plurality of incremental blocks to generate a second sha256 set; Obtain the third sha256 set and the second data offset corresponding to the parent backup shard block data, and the third data offset of the target shard block data; In response to detecting that the second sha256 set is different from the third sha256 set, and / or the second data offset is different from the third data offset, recording the third data offset and the second sha256 set of the target shard block data, and compressing the target shard block data; The compressed target shard block data is transferred and backed up.

8. A data processing device, It is characterized in that The device comprises: A data acquisition module is used to acquire data that needs to be backed up when a data backup request is detected; A concurrent number determination module, used to determine a target concurrent number based on a host machine memory and a host machine network card performance, wherein the host machine is a host machine where the data to be backed up is located; A shard block data acquisition module, used to split the data to be backed up according to the target concurrency number to obtain at least one shard block data; A backup mode determination module, used to determine the backup mode of the target shard block data based on the number of backups of the target shard block data, wherein the backup mode is full backup or incremental backup; The transmission backup module is used to perform multi-threaded transmission backup of multiple shard block data according to the backup mode and the target concurrency number.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Data preservation method for server and application system

    CN120315946A