Data scanning methods, electronic devices, storage media and software products

Abstract

This application provides a data scanning method, electronic device, storage medium, and program product, relating to the field of data storage technology. The method includes: obtaining the number of objects included in the data to be scanned in an object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object; determining the number of scanning tasks and the at least one storage sub-unit corresponding to each scanning task based on the number of objects; and allocating multiple scanning tasks to multiple worker nodes in a distributed system for execution based on the number of tasks and the at least one storage sub-unit corresponding to each scanning task. In this application's embodiments, dynamic and scalable task generation is achieved; allocating multiple scanning tasks to multiple worker nodes for execution utilizes the computing power of distributed computing, improving data scanning efficiency and avoiding the problem of excessive CPU utilization in single-machine scenarios, thus improving the running performance of data scanning tasks.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data scanning method, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of cloud computing technology, cloud storage capacity has grown rapidly, and bandwidth capabilities during data migration have also improved rapidly. 10 gigabits per second (Gigabit / s) bandwidth links are gradually becoming common in cloud object storage, and physical link networks are gradually becoming capable of reaching 100 Gigabit / s. Against this backdrop, cloud object storage data migration is also gradually setting its sights on 100 Gigabit / s.

[0003] During data migration, bandwidth gradually ceases to be the bottleneck; instead, the ability to scan source storage units becomes the bottleneck. Object storage offers significant advantages when handling large amounts of unstructured data, but this presents challenges for the data migration process. For example, assuming the source storage space can scan 50,000 objects per second, and each file is 1KB, the replication bandwidth would only reach approximately 48Mb, far below the 10Gigabit / s or 100Gigabit / s bandwidth provided by the network link. In this scenario, source-side data scanning severely hinders the progress of data migration. Summary of the Invention

[0004] This application provides a data scanning method, electronic device, storage medium, and program product to solve the problem that slow source data scanning affects data migration speed.

[0005] In a first aspect, embodiments of this application provide a data scanning method, comprising: obtaining the number of objects included in the data to be scanned in an object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object; determining the number of scanning tasks and the at least one storage sub-unit corresponding to each scanning task based on the number of objects; and allocating multiple scanning tasks to multiple worker nodes in a distributed system for execution based on the number of tasks and the at least one storage sub-unit corresponding to each scanning task.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0008] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the methods described above.

[0009] Compared with the prior art, this application has the following advantages:

[0010] This application provides a data scanning method, electronic device, storage medium, and program product. First, the number of objects included in the data to be scanned in the object storage unit is obtained. The data to be scanned includes at least one storage sub-unit, and each storage sub-unit includes at least one object. Then, based on the number of objects, the number of scanning tasks and the at least one storage sub-unit corresponding to each scanning task are determined. Based on the number of tasks and the at least one storage sub-unit corresponding to each scanning task, multiple scanning tasks are allocated to multiple worker nodes in a distributed system for execution. In this embodiment, scanning tasks can be dynamically divided according to the number of objects, realizing dynamic and scalable task generation. Allocating multiple scanning tasks to multiple worker nodes for execution utilizes the computing power of distributed computing, improving data scanning efficiency and avoiding the problem of excessive CPU utilization in single-machine scenarios, thus improving the running performance of data scanning tasks.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of a data scanning method according to an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of thread concurrency control in a data scanning method according to an embodiment of this application.

[0015] Figure 3 This is a schematic diagram illustrating the task distribution of a data scanning method according to an embodiment of this application.

[0016] Figure 4 This is a flowchart of a data scanning method according to an embodiment of this application.

[0017] Figure 5 This is a flowchart of a data scanning method according to an embodiment of this application.

[0018] Figure 6 This is a structural block diagram of a data scanning device according to an embodiment of this application.

[0019] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0023] Data migration refers to the automatic and asynchronous replication of object creation, updating, and deletion operations from a source storage unit to a target storage space (bucket) to achieve compliance, reduce latency, and ensure security and availability. Data migration can move data from one storage unit to another in cloud storage products, meeting users' disaster recovery or data replication needs. When migrating historical data from a storage unit, it is necessary to scan the original storage unit's data, i.e., enumerate information about all objects in the storage space. In related technologies, the process of scanning the original storage unit's data involves a single machine performing a fixed number of scan tasks. With the rapid growth of data volume, this method has gradually become inefficient and slow, affecting the speed of data migration.

[0024] In this embodiment, the number of scanning tasks and at least one storage sub-unit corresponding to each scanning task are determined according to the number of objects. The scanning tasks can be dynamically divided according to the number of objects, realizing the dynamic and scalable generation of tasks. Multiple scanning tasks are distributed to multiple working nodes in the distributed system for execution, which utilizes the computing power of distributed computing, improves the data scanning efficiency, and avoids the problem of excessive CPU utilization in single-machine cases, thereby improving the running performance of data scanning tasks.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of the data scanning method provided in this application. In this embodiment, the worker node is a device that executes data scanning tasks. The object storage unit (bucket) includes multiple storage partitions. Each scanning task corresponds to a scanning range, each scanning range includes at least one storage partition, and each storage partition includes at least one object. A key-value pair storage engine (KV engine) is used to manage the storage partitions. The KV engine stores metadata for each storage partition, such as which objects each storage partition includes. Worker node 1 obtains the number of objects included in each storage partition in the data to be scanned from the KV engine, and determines the number of scanning tasks and the storage partitions corresponding to each scanning task based on the number of objects. Figure 1 As shown, two scanning tasks are defined: Scanning Task 1 corresponds to Scan Range 1, which includes Storage Sub-unit 1; Scanning Task 2 corresponds to Scan Range 2, which includes Storage Sub-units 2 and 3. Storage Sub-units 1, 2, and 3 do not overlap. The scanning tasks are distributed to different worker nodes for execution using a hash function. Specifically, the hash value of the task identifier for each scanning task is calculated. Based on the hash value of each scanning task, the corresponding worker node for each scanning task is determined. Scan Range 1 of Scanning Task 1 is assigned to worker node 2 for execution, and Scan Range 2 of Scanning Task 2 is assigned to worker node 3 for execution via Remote Procedure Call (RPC).

[0026] Figure 2 This diagram illustrates thread concurrency control for the data scanning method provided in this application. To avoid excessive pressure on the KV engine, a distributed lock is used to control the number of threads accessing the KV engine; only threads holding the distributed lock can access the KV engine. Acquiring the distributed lock (e.g., ...) Figure 2 The threads of "Lock A" and "Lock B" shown (e.g.) Figure 2The "working node" (shown) calls the KV engine's interface function to obtain the number of objects in each storage sub-unit (e.g., the number of object keys), calculates the sum of the object counts in each storage sub-unit, and divides each storage sub-unit into different scan ranges. A threshold for the number of objects is set for each scan task. When the sum of the number of objects in the storage sub-units within a scan range equals the threshold, adding storage sub-units to that scan range stops. To prevent an excessive number of tasks from being generated, a task count threshold is set for each scan task. When the number of tasks exceeds the threshold, a merging operation is performed on the scan tasks. Using a greedy algorithm, the scan task with the smallest sum of object counts (corresponding to the scan range) is selected each time and merged into a single scan task, such as... Figure 2 As shown, scan range 1 and scan range 2 are merged to obtain the merged scan range 1, thereby making the number of objects in each scan task as close as possible.

[0027] Figure 3 This diagram illustrates the task distribution process of the data scanning method provided in this application. During task distribution, a hash function is used to scatter the data, calculate the hash value corresponding to the task identifier of each task, determine the corresponding worker node for each task, and then distribute the tasks to the appropriate worker nodes. For example... Figure 3 As shown, scan task 1 (corresponding to scan range 1) and scan task 2 (corresponding to scan range 2) are randomly distributed to multiple devices S1, S2, S3, and S4 via RPC requests for execution. The worker node executing the scan task remains unchanged during task execution, ensuring that even if the task is interrupted for some reason, the worker node executing the scan task can still be found based on the hash value upon restarting, and the interrupted scan task can be resumed using that worker node. Additionally, during task execution, task progress information is recorded; specifically, scan progress checkpoints are written to the KV engine to minimize repetitive listing operations and to allow the scan task to resume from the interrupted checkpoint if it is interrupted due to upgrades or other reasons. During task execution, the worker node distributing the task periodically sends RPC requests to each worker node executing the scan task to check whether each scan task has finished. Figure 3 As shown, check whether the scanning tasks in S3 and S4 are completed. When all scanning tasks are completed, the entire scanning task is considered to be finished.

[0028] This application provides a data scanning method. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0029] like Figure 4 The diagram shown is a flowchart of a data scanning method according to an embodiment of this application, including:

[0030] Step S401: Obtain the number of objects included in the data to be scanned in the object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object.

[0031] Object storage units are units that store data using object storage methods, such as buckets. Object storage processes, stores, and retrieves data as objects. It manages data as objects, each containing the data itself, metadata, and a globally unique identifier. Object storage offers significant advantages when handling large amounts of unstructured data.

[0032] The data to be scanned can be data that needs to be scanned during the data migration process, including at least one storage partition, and each storage partition includes at least one object.

[0033] In practical applications, the specific number of objects included in each storage subunit can be obtained by calling the interface function.

[0034] Step S402: Determine the number of scanning tasks and at least one storage sub-unit corresponding to each scanning task based on the number of objects.

[0035] The number of objects in the data to be scanned is the total number of objects. A task number threshold is set. Based on the total number of objects, and provided the number of tasks does not exceed the task number threshold, the number of tasks and the number of objects corresponding to each task are determined by distributing objects as evenly as possible among each scan task. Based on the number of objects corresponding to each task and the number of objects included in each storage sub-unit, each storage sub-unit is allocated to scan tasks. However, the number of objects included in each storage sub-unit may differ; therefore, the final number of objects corresponding to each scan task may vary.

[0036] Step S403: Based on the number of tasks and at least one storage sub-unit corresponding to each scanning task, distribute the multiple scanning tasks to multiple worker nodes in the distributed system for execution.

[0037] After the task is divided, the scanning task is assigned to multiple worker nodes to perform the scan. The information of the objects in the storage sub-units corresponding to the scanning task is listed. The efficiency of data scanning is improved by using distributed execution.

[0038] The data scanning method provided in this application first obtains the number of objects included in the data to be scanned in the object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object; then, based on the number of objects, the number of scanning tasks and the at least one storage sub-unit corresponding to each scanning task are determined; based on the number of tasks and the at least one storage sub-unit corresponding to each scanning task, multiple scanning tasks are allocated to multiple worker nodes in a distributed system for execution. In this application embodiment, scanning tasks can be dynamically divided according to the number of objects, realizing dynamic and scalable task generation; allocating multiple scanning tasks to multiple worker nodes for execution utilizes the computing power of distributed computing, improves data scanning efficiency, and avoids the problem of excessive CPU utilization in single-machine cases, thereby improving the running performance of data scanning tasks.

[0039] The following describes the specific implementation process of each step above through various implementation methods:

[0040] In one implementation, step S402, determining the number of scanning tasks and at least one storage sub-unit corresponding to the scanning task based on the number of objects, includes: step S4021, determining the number of tasks based on the number of objects and the object number threshold corresponding to the scanning task; step S4022, allocating at least one storage sub-unit to the scanning task based on the number of objects included in each storage sub-unit of the data to be scanned and the number of tasks, wherein the number of objects corresponding to the scanning task does not exceed the object number threshold.

[0041] In practical applications, scanning tasks are divided according to the number of objects, and storage sub-units are assigned to scanning tasks. If there are more objects, more scanning tasks are assigned.

[0042] A threshold for the number of objects corresponding to a scan task is preset. When dividing scan tasks, the number of objects in a scan task is guaranteed to be less than or equal to the threshold. For any scan task, once the number of objects corresponding to that scan task equals the threshold, no more storage sub-units are allocated to that scan task, thus ensuring a balanced number of objects across scan tasks.

[0043] In one example, the number of objects in the data to be scanned is 100, and the threshold for the number of objects corresponding to each scan task is 21. Therefore, based on the fact that each scan task corresponds to 20 objects, multiple storage sub-units can be divided into 5 scan tasks, and the number of objects in each scan task should be as balanced as possible and not exceed 21.

[0044] In one implementation, after determining the number of tasks based on the number of objects and the object number threshold corresponding to the scanning task in step S4021, the data scanning method further includes: if the number of tasks exceeds the task number threshold, then determining at least two scanning tasks with the fewest objects after addition among multiple scanning tasks, and merging the at least two scanning tasks with the fewest objects after addition to obtain a merged task.

[0045] A pre-set threshold for the number of tasks is set. When dividing tasks, the number of tasks must be less than or equal to the threshold. If the number of tasks exceeds the threshold, at least two scan tasks with the fewest objects when added together are obtained and merged to prevent the final number of tasks from being too large and to ensure that the number of objects in each scan task is as close as possible.

[0046] Optionally, a greedy algorithm can be used to find the two scan tasks with the fewest objects when added together, and then merge these two scan tasks.

[0047] In one implementation, step S401, obtaining the number of objects included in the data to be scanned in the object storage unit, includes: when multiple threads need to obtain the number of objects, using the thread holding the distributed lock among the multiple threads to obtain the number of objects.

[0048] Distributed locks are used to coordinate access to shared resources by multiple threads in a distributed system. To avoid the pressure caused by multiple thread operations and to prevent tasks from being executed repeatedly, distributed locks are used for concurrency control. Only a predetermined number of threads can successfully acquire the distributed lock at the same time. Only the thread holding the distributed lock can call the interface function to get the number of objects. After the thread holding the lock completes its operation, it releases the distributed lock, which is then made available for use by other waiting threads.

[0049] In one implementation, step S401, obtaining the number of objects included in the data to be scanned in the object storage unit, includes: obtaining the number of objects included in each storage sub-unit of at least one storage sub-unit included in the data to be scanned; summing the number of objects included in each storage sub-unit to obtain the total number of objects.

[0050] In practical applications, the list of storage sub-units included in the data to be scanned is obtained by calling the interface function, and the number of objects included in each storage sub-unit is calculated, thereby obtaining the number of objects included in the data to be scanned.

[0051] In one example, the GetPartitionsByRange function is used to obtain a list of storage sub-cells, and the GetPartitionSum function is used to obtain the sum of the number of objects included in each storage sub-cell. The sum of the number of objects in each storage sub-cell is calculated to obtain the number of objects included in the data to be scanned.

[0052] In this embodiment, the number of objects in the data to be scanned is obtained by calculating the number of objects included in each storage sub-unit. The calculation process is simple and the calculation results are accurate.

[0053] In one implementation, step S403, which involves distributing multiple scanning tasks to multiple worker nodes in a distributed system for execution based on the number of tasks and at least one storage sub-unit corresponding to each scanning task, includes: step S4031, determining the number of worker nodes based on the number of tasks; and step S4032, distributing at least one storage sub-unit corresponding to each scanning task to the number of worker nodes for execution.

[0054] In practical applications, the number of worker nodes can be determined based on the number of tasks. Each scan task can be executed by one worker node, or multiple scan tasks can be executed by one worker node. After determining the number of nodes, the scan tasks are distributed to multiple worker nodes for execution, thereby achieving distributed execution of scan tasks.

[0055] In this embodiment, the number of working nodes is determined based on the number of tasks, and then tasks are allocated. By using multiple working nodes to execute tasks, the efficiency of task execution can be improved.

[0056] In one implementation, step S4032, which assigns at least one storage sub-unit corresponding to each scanning task to a number of working nodes for execution, includes: assigning at least one storage sub-unit corresponding to each scanning task to a number of working nodes for execution via RPC.

[0057] Remote Procedure Call (RPC) is a protocol that allows a program to execute a subroutine or function on another computer on a network without the programmer explicitly coding the underlying network communication details.

[0058] In this embodiment, scanning tasks are distributed via RPC. Different systems can interact through the standardized RPC protocol, enabling task distribution even if they are written in different programming languages ​​or run on different operating systems. Furthermore, RPC supports asynchronous call modes, allowing the request sender to continue executing other tasks after issuing a request without waiting for a response from the receiver, thus improving concurrency capabilities.

[0059] In one implementation, step S4032, which involves allocating at least one storage sub-unit corresponding to each scan task to a number of working nodes for execution, includes: performing hash calculations on the task identifiers of each scan task to obtain hash values ​​corresponding to each scan task; determining the working nodes corresponding to each scan task based on the hash values ​​corresponding to each scan task; and allocating at least one storage sub-unit corresponding to each scan task to a number of working nodes for execution based on the working nodes corresponding to each scan task.

[0060] The task identifier is a string that uniquely identifies the scanning task. The task identifier is hashed using a hash function (such as MD5) to obtain the hash value corresponding to each scanning task. Multiple hash values ​​are then mapped to multiple worker nodes.

[0061] In one example, the number of worker nodes is N. For each scan task, a hash function is used to hash the task identifier, resulting in a fixed-length hash value. The resulting hash value is then converted into a non-negative integer (e.g., using a modulo operation). This non-negative integer is then modulo the number of worker nodes N, resulting in an integer between 0 and N-1. Based on this integer, the scan task is assigned to the corresponding worker node.

[0062] In this embodiment, a hash function is used to calculate which worker node should execute each task, which can ensure that the scanning tasks are evenly distributed across the worker nodes during allocation, avoiding overloading of a single worker node.

[0063] In one implementation, the data scanning method further includes: if any scanning task is interrupted, determining the target working node corresponding to the interrupted task based on the hash value corresponding to the interrupted task; and using the target working node to execute the interrupted scanning task.

[0064] In practical applications, the worker node executing the scanning task will not change during the task execution process, ensuring that even if the task is interrupted for some reason, the worker node executing the scanning task can still be found based on the hash value when the task is restarted, and the interrupted scanning task can be continued using that worker node.

[0065] In one implementation, the data scanning method further includes: recording task execution progress information during the execution of the scanning task; determining the task interruption point based on the task execution progress information if the scanning task is interrupted; and continuing to execute the remaining scanning task from the task interruption point.

[0066] During task execution, progress information is recorded, for example, using a checkpoint mechanism. At preset time intervals or event triggers, such as once per hour or at the end of each task, the task execution status is recorded, including but not limited to variables in memory, file pointer positions, transaction logs, and the queue of unfinished operations, and written to persistent storage. Based on the recorded task execution status, the task interruption point is determined. If a task is interrupted due to upgrades or other reasons, the remaining scanning tasks resume from the point of interruption, avoiding repetitive execution from the beginning and improving task execution efficiency.

[0067] This application provides a data scanning method. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0068] like Figure 5 The diagram shown is a flowchart of a data scanning method according to an embodiment of this application, including:

[0069] Step S501: Obtain the number of objects included in each storage sub-unit of at least one storage sub-unit of the data to be scanned.

[0070] Step S502: Sum the number of objects included in each storage sub-unit to obtain the total number of objects.

[0071] The GetPartitionsByRange function is used to obtain the list of storage sub-cells, and the GetPartitionSum function is used to obtain the sum of the number of objects included in each storage sub-cell. The sum of the number of objects in each storage sub-cell is calculated to obtain the number of objects included in the data to be scanned.

[0072] Step S503: Determine the number of tasks based on the number of objects and the threshold number of objects corresponding to the scanning task.

[0073] Scanning tasks are divided based on the number of objects. Storage sub-units are assigned to scan tasks, with more scan tasks assigned to a larger number of objects. A threshold for the number of objects in each scan task is pre-set, ensuring that the number of objects in a scan task is less than or equal to the threshold when dividing scan tasks. For any scan task, if the number of objects in that scan task exceeds the threshold, no more storage sub-units will be assigned to that scan task, thus ensuring a balanced number of objects across scan tasks.

[0074] Step S504: Based on the number of objects and the number of tasks included in each storage sub-unit of the data to be scanned, at least one storage sub-unit is assigned to a scanning task.

[0075] The number of objects corresponding to a scanning task shall not exceed the object number threshold.

[0076] Step S505: Determine the number of working nodes based on the number of tasks.

[0077] The number of worker nodes can be determined based on the number of tasks. Each scan task can be executed by one worker node, or multiple scan tasks can be executed by one worker node. After determining the number of nodes, the scan tasks are distributed to multiple worker nodes for execution, thereby achieving distributed execution of scan tasks.

[0078] Step S506: Perform hash calculation on the task identifier of each scanning task to obtain the hash value corresponding to each scanning task.

[0079] The task identifier is a string that uniquely identifies the scanning task. A hash function (such as MD5) is used to perform a hash calculation on the task identifier to obtain the hash value corresponding to each scanning task.

[0080] Step S507: Determine the working node corresponding to each scanning task based on the hash value corresponding to each scanning task.

[0081] Specifically, the number of worker nodes is N. For each scan task, a hash function is used to hash the task identifier, resulting in a fixed-length hash value. The hash value is then converted into a non-negative integer (e.g., using a modulo operation). This non-negative integer is then modulo the number of worker nodes N, resulting in an integer between 0 and N-1. Based on this integer, the scan task is assigned to the corresponding worker node.

[0082] Step S508: Based on the working nodes corresponding to each scanning task, allocate at least one storage sub-unit corresponding to each scanning task to the number of working nodes for execution.

[0083] Step S509: During the scanning task execution process, record the task execution progress information.

[0084] Step S510: In the event of a task interruption, determine the task interruption point based on the task execution progress information.

[0085] Step S511: Continue executing the remaining scanning task from the point where the task was interrupted.

[0086] During task execution, progress information is recorded, for example, using a checkpoint mechanism. At preset time intervals or event triggers, such as once per hour or at the end of each task, the task execution status is recorded, including but not limited to variables in memory, file pointer positions, transaction logs, and the queue of unfinished operations, and written to persistent storage. Based on the recorded task execution status, the task interruption point is determined. If a task is interrupted due to upgrades or other reasons, the remaining scanning tasks resume from the point of interruption, avoiding repetitive execution from the beginning and improving task execution efficiency.

[0087] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data scanning device. For example... Figure 6 The diagram shown is a structural block diagram of a data scanning device according to an embodiment of this application. The device includes:

[0088] The acquisition module 601 is used to acquire the number of objects included in the data to be scanned in the object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object.

[0089] The determination module 602 is used to determine the number of scanning tasks and at least one storage sub-unit corresponding to each scanning task based on the number of objects.

[0090] The allocation module 603 is used to allocate multiple scanning tasks to multiple worker nodes in the distributed system for execution based on the number of tasks and at least one storage sub-unit corresponding to each scanning task.

[0091] The data scanning apparatus provided in this application first obtains the number of objects included in the data to be scanned in the object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object; then, based on the number of objects, the number of scanning tasks and the at least one storage sub-unit corresponding to each scanning task are determined; based on the number of tasks and the at least one storage sub-unit corresponding to each scanning task, multiple scanning tasks are allocated to multiple worker nodes in a distributed system for execution. In this application embodiment, scanning tasks can be dynamically divided according to the number of objects, realizing dynamic and scalable task generation; allocating multiple scanning tasks to multiple worker nodes for execution utilizes the computing power of distributed computing, improves data scanning efficiency, and avoids the problem of excessive CPU utilization in single-machine cases, thereby improving the running performance of data scanning tasks.

[0092] In one implementation, the determining module 602 is used to: determine the number of tasks based on the number of objects and the object number threshold corresponding to the scanning task; and allocate at least one storage sub-unit to the scanning task based on the number of objects included in each storage sub-unit of the data to be scanned and the number of tasks, wherein the number of objects corresponding to the scanning task does not exceed the object number threshold.

[0093] In one implementation, the data scanning device further includes: after determining the number of tasks based on the number of objects and the object number threshold corresponding to the scanning task, if the number of tasks exceeds the task number threshold, then determining at least two scanning tasks among the multiple scanning tasks that have the fewest objects when added together, and merging the at least two scanning tasks with the fewest objects when added together to obtain a merged task.

[0094] In one implementation, the acquisition module 601 is used to: when multiple threads need to acquire the number of objects, acquire the number of objects by utilizing the thread holding the distributed lock among the multiple threads.

[0095] In one implementation, the acquisition module 601 is used to: acquire the number of objects included in each storage sub-unit of at least one storage sub-unit of the data to be scanned; and sum the number of objects included in each storage sub-unit to obtain the total number of objects.

[0096] In one implementation, the allocation module 603 is used to: determine the number of working nodes based on the number of tasks; and allocate at least one storage sub-unit corresponding to each scanning task to the number of working nodes for execution.

[0097] In one implementation, when allocating at least one storage sub-unit corresponding to each scan task to a number of working nodes for execution, the allocation module 603 is used to: allocate at least one storage sub-unit corresponding to each scan task to a number of working nodes for execution via remote procedure call.

[0098] In one implementation, when allocating at least one storage sub-unit corresponding to each scan task to a number of worker nodes for execution, the allocation module 603 performs the following: hash calculations on the task identifiers of each scan task to obtain hash values ​​corresponding to each scan task; determines the worker nodes corresponding to each scan task based on the hash values ​​corresponding to each scan task; and allocates at least one storage sub-unit corresponding to each scan task to a number of worker nodes for execution based on the worker nodes corresponding to each scan task.

[0099] In one implementation, the data scanning device is further configured to: determine the target working node corresponding to the interrupted task based on the hash value corresponding to the interrupted task in the event that any of the scanning tasks is interrupted; and execute the interrupted scanning task using the target working node.

[0100] In one implementation, the data scanning device is further configured to: record task execution progress information during the execution of the scanning task; determine the task interruption point based on the task execution progress information in the event of a task interruption; and continue executing the remaining scanning task from the task interruption point.

[0101] The functions of each module in the embodiments of this application can be found in the corresponding descriptions in the above methods, and they have corresponding beneficial effects, which will not be repeated here.

[0102] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 7 As shown, the electronic device includes a memory 710 and a processor 720. The memory 710 stores a computer program that can run on the processor 720. When the processor 720 executes the computer program, it implements the method described in the above embodiments. The number of memories 710 and processors 720 can be one or more.

[0103] The electronic device also includes:

[0104] The communication interface 730 is used to communicate with external devices and perform data exchange and transmission.

[0105] If the memory 710, processor 720, and communication interface 730 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 710, processor 720, and communication interface 730 are integrated on a single chip, then the memory 710, processor 720, and communication interface 730 can communicate with each other through an internal interface.

[0107] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0108] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the method provided in this application.

[0109] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0110] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0111] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware engines, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0112] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0116] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0117] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0118] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0120] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data scanning method, characterized in that, include: Obtain the number of objects included in the data to be scanned in the object storage unit; the data to be scanned includes at least one storage sub-unit, and the storage sub-unit includes at least one object; Based on the number of objects, determine the number of scanning tasks and at least one storage sub-unit corresponding to each scanning task; Based on the number of tasks and at least one storage sub-unit corresponding to each scanning task, the multiple scanning tasks are allocated to multiple worker nodes in the distributed system for execution.

2. The method according to claim 1, characterized in that, The step of determining the number of scanning tasks and at least one storage sub-unit corresponding to the scanning task based on the number of objects includes: The number of tasks is determined based on the number of objects and the threshold number of objects corresponding to the scanning task; Based on the number of objects included in each storage sub-unit of the data to be scanned and the number of tasks, at least one storage sub-unit is assigned to the scanning task, wherein the number of objects corresponding to the scanning task does not exceed the object number threshold.

3. The method according to claim 2, characterized in that, After determining the number of tasks based on the number of objects and the object number threshold corresponding to the scanning task, the method further includes: If the number of tasks exceeds the task number threshold, then at least two scanning tasks with the fewest objects after addition are identified from the multiple scanning tasks, and the at least two scanning tasks with the fewest objects after addition are merged to obtain a merged task.

4. The method according to claim 1, characterized in that, The number of objects included in the data to be scanned in the object storage unit includes: When multiple threads need to obtain the number of objects, the thread holding the distributed lock among the multiple threads is used to obtain the number of objects.

5. The method according to any one of claims 1-4, characterized in that, The number of objects included in the data to be scanned in the object storage unit includes: Obtain the number of objects included in each storage sub-unit of at least one storage sub-unit of the data to be scanned; The number of objects is obtained by summing the number of objects included in each storage subunit.

6. The method according to any one of claims 1-4, characterized in that, The step of allocating multiple scanning tasks to multiple worker nodes in a distributed system for execution based on the number of tasks and at least one storage sub-unit corresponding to each scanning task includes: The number of work nodes is determined based on the number of tasks. At least one storage sub-unit corresponding to each scanning task is assigned to the specified number of working nodes for execution.

7. The method according to claim 6, characterized in that, The step of allocating at least one storage sub-unit corresponding to each scanning task to the specified number of worker nodes for execution includes: By means of remote procedure calls, at least one storage sub-unit corresponding to each scanning task is allocated to the number of working nodes for execution.

8. The method according to claim 6, characterized in that, The step of allocating at least one storage sub-unit corresponding to each scanning task to the specified number of worker nodes for execution includes: The task identifier of each scanning task is hashed to obtain the hash value corresponding to each scanning task. The working node corresponding to each scanning task is determined based on the hash value corresponding to each scanning task. Based on the working nodes corresponding to each scanning task, at least one storage sub-unit corresponding to each scanning task is allocated to the specified number of working nodes for execution.

9. The method according to claim 8, characterized in that, The method further includes: If any of the scanning tasks is interrupted, the target working node corresponding to the interrupted task is determined based on the hash value corresponding to the interrupted task. The interrupted scanning task is executed using the target working node.

10. The method according to any one of claims 1-4, characterized in that, The method further includes: During the scanning task execution process, record the task execution progress information; In the event of a task interruption, the task interruption point is determined based on the task execution progress information. Continue executing the remaining scanning task from the point where the task was interrupted.

11. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-10.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1-10.

Images