Data scanning method and device, electronic device, storage medium, and program product

Abstract

The embodiments of the present application disclose a data scanning method and device, electronic device, storage medium, and program product, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: parsing a scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in a storage system; wherein the storage bucket includes a carrier for object storage; dividing the storage area of the to-be-scanned storage bucket into multiple to-be-scanned sub-areas, and splitting the scanning task into multiple scanning subtasks based on the multiple to-be-scanned sub-areas obtained by the division; and executing the split scanning subtasks to query the target object in the corresponding to-be-scanned sub-areas. The technical solution of the embodiments of the present application can improve scanning speed and success rate.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data scanning method and device, electronic equipment, storage medium, and program product. Background Art

[0002] A storage system includes buckets, which store objects. Objects are the basic unit of object storage and can be understood as data of any format or type, such as images, documents, audio, and video files. To facilitate user management of objects stored in buckets, buckets can be scanned based on user configuration. However, in related technologies, bucket scanning is slow and has a high failure rate. Summary of the Invention

[0003] To solve the above technical problems, embodiments of the present application provide a data scanning method and device, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of an embodiment of the present application, a data scanning method is provided, the method comprising:

[0005] Parsing the scan task to determine a to-be-scanned storage bucket corresponding to the scan task in a storage system; wherein the storage bucket includes a carrier for object storage;

[0006] Dividing the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and splitting the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division;

[0007] The split scanning subtasks are executed to query the target object from the corresponding sub-area to be scanned.

[0008] According to one aspect of an embodiment of the present application, a data scanning device is provided, the device comprising:

[0009] A parsing module configured to parse the scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in the storage system; wherein the storage bucket includes a carrier of object storage;

[0010] a splitting module configured to divide the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and to split the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division;

[0011] The execution module is configured to execute the split scanning subtasks to query the target object from the corresponding sub-area to be scanned.

[0012] According to one aspect of an embodiment of the present application, an electronic device is provided, including:

[0013] one or more processors;

[0014] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the data scanning method as described above.

[0015] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of an electronic device, the electronic device executes the data scanning method as described above.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, wherein when the computer instructions are executed by a processor, the data scanning method as described above is implemented.

[0017] In the technical solution provided in the embodiments of the present application, on the one hand, the scanning task is split into multiple scanning sub-tasks, and each scanning sub-task is executed, thereby improving the scanning speed and success rate; on the other hand, the storage area of the storage bucket to be scanned is divided into multiple sub-areas to be scanned, and the corresponding sub-areas to be scanned are scanned when executing the scanning sub-task, which can avoid repeated scanning.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0020] Figure 1 It is a schematic diagram of an implementation environment involved in this application;

[0021] Figure 2 is a flow chart of a data scanning method shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a global index table shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a scanning task splitting process shown in an exemplary embodiment of the present application;

[0024] Figure 5 yes Figure 2 A flow chart of step S130 in the illustrated embodiment in an exemplary embodiment;

[0025] Figure 6 yes Figure 2 A flow chart of step S130 in the illustrated embodiment in an exemplary embodiment;

[0026] Figure 7 yes Figure 6 A flow chart of step S320 in the illustrated embodiment in an exemplary embodiment;

[0027] Figure 8 is a block diagram illustrating a data scanning process according to an exemplary embodiment of the present application;

[0028] Figure 9 is a flowchart of a subtask scheduling process shown in an exemplary embodiment of the present application;

[0029] Figure 10 is a structural diagram of a data scanning device shown in an exemplary embodiment of the present application;

[0030] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0034] It should also be noted that the term "plurality" used in this application refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0035] A storage system includes buckets, each containing objects. To facilitate user management of objects stored in the buckets, buckets can be scanned based on user configuration. In related art, when performing a bucket scan task, the scanning tasks for all buckets are processed as a whole. However, the large amount of data stored in the buckets results in slow scanning speeds and high task failure rates. Based on this, embodiments of the present application provide a data scanning method and apparatus, electronic device, storage medium, and program product that can improve scanning speeds and reduce failure rates.

[0036] See also Figure 1 , Figure 1 Schematic diagram of an implementation environment involved in this application, which includes a data scanning device 100 and a storage system 200. The storage system 200 includes multiple storage buckets. The data scanning device 100 and the storage system 200 communicate with each other via a wired or wireless network.

[0037] It should be understood that Figure 1 The number of data scanning devices 100, storage systems 200, and storage buckets in the embodiment is merely illustrative. Any number of data scanning devices 100, storage systems 200, and storage buckets may be provided according to actual needs.

[0038] The data scanning device 100 can be a server or other device. The server can be a server that provides various services. It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This is not limited here.

[0039] Storage system 200 can be a storage system based on cloud storage technology, or other types of storage systems. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system is a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (also known as storage nodes) on a network through application software or application interfaces to work together and provide external data storage and service access capabilities.

[0040] Currently, the storage method of distributed cloud storage systems is to create logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may be composed of disks on a storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into many parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID, IDentity). The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0041] The distributed cloud storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) grouping. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0042] Bucket is a carrier for object storage, which can be understood as a "container" for storing objects, and the "container" can have no upper limit on capacity. Objects can be stored in a bucket in a flat structure, without the concept of folders and directories. Users can choose to store objects in a single or multiple buckets. The user can initiate a request to create a bucket in the storage system 200 so that the storage system 200 creates a bucket according to the request. The user can also configure the created bucket, for example, configure at least one of the name of the bucket, the region to which the bucket belongs, and the access rights to the bucket. The region to which the bucket belongs is the storage location of the bucket. For example, the bucket can be deployed on a storage node located in Beijing, and Beijing is the region to which the bucket belongs.

[0043] In an exemplary embodiment, the data scanning method provided in the embodiment of the present application can be executed by a data scanning device 100. The data scanning device 100 can obtain a scanning task and parse the scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in a storage system; divide the storage area of the to-be-scanned storage bucket into multiple to-be-scanned sub-areas, and split the scanning task into multiple scanning sub-tasks based on the multiple to-be-scanned sub-areas obtained by the division; execute the split scanning sub-tasks to query the target object from the corresponding to-be-scanned sub-areas. On the one hand, splitting the scanning task into multiple scanning sub-tasks and executing each scanning sub-task improves the scanning speed and success rate; on the other hand, dividing the area corresponding to the to-be-scanned storage bucket into multiple to-be-scanned sub-areas and scanning the corresponding to-be-scanned sub-areas when executing the scanning sub-tasks can avoid repeated scanning and achieve parallel scanning within the storage bucket, further improving the scanning speed.

[0044] The data scanning device 100 may obtain scanning tasks from user terminals, which include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals.

[0045] It should be noted that, in addition to the aforementioned application scenarios, the embodiments of the present application can also be applied to various application scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. For example, if applied to smart transportation or assisted driving scenarios, the data stored in the storage system 200 may include maps and other data related to smart transportation or assisted driving, and the user terminal may be an in-vehicle terminal, a navigation terminal, etc.

[0046] See also Figure 2 , Figure 2 This is a flow chart of a data scanning method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 The implementation environment shown, which can be Figure 1 The data scanning device 100 is executed in the illustrated implementation environment.

[0047] like Figure 2 As shown, in an exemplary embodiment, the data scanning method may include steps S110 to S130, which are described in detail as follows:

[0048] Step S110 , parsing the scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in the storage system; wherein the storage bucket includes a carrier for object storage.

[0049] It should be noted that a storage system refers to a computer system consisting of various storage devices that store programs and data, control components, and equipment (hardware) and algorithms (software) for managing information scheduling. A storage system can be Cloud Object Storage (COS), a distributed storage service, or other storage systems.

[0050] The storage system contains multiple buckets. Buckets are the carriers of object storage and can be understood as "containers" for storing objects. These "containers" can have unlimited capacity. Objects can be stored in buckets in a flat structure, without the concept of folders and directories. Users can choose to store objects in a single bucket or multiple buckets. The storage system can provide a bucket creation service. Users can use this service to initiate a bucket creation request to the storage system, which will create a bucket based on the request. Users can also configure the bucket, for example, to configure at least one of the following: the bucket name, the region to which the bucket belongs, and the bucket's access permissions.

[0051] A scan task is used to instruct a user to scan the buckets contained in a storage system to retrieve target objects. The user can configure the scan task based on actual needs, for example, configuring the bucket to be scanned, the scanning method, etc. Scanning methods include, but are not limited to, periodic scanning and real-time scanning. Periodic scanning is executing the scan task at regular intervals, and real-time scanning is executing the scan task in real time after receiving the scan task. The specific form of the scan task can be flexibly set according to actual needs. In one example, the scan task can be an inventory task. Based on the configuration information of the inventory task, it can scan specified objects in the bucket or objects with the same object prefix and output an inventory report. For example, the scan task can be scheduled daily or weekly to scan specified objects in the bucket or objects with the same object prefix and output an inventory report, making it easier for users to manage objects in the bucket. The inventory report can include the stored objects and their corresponding metadata, and based on the user's configuration, it can also include the object attribute information required by the user. The format of the inventory report can be flexibly set according to actual needs. For example, it can be CSV (Comma-Separated Values) format, of course, other formats are also possible. During the inventory task execution, you can scan only attribute information such as object metadata without reading the object content.

[0052] The number of buckets to be scanned corresponding to a scanning task may be multiple; considering that different buckets have different access permissions, in order to facilitate permission control, the number of buckets to be scanned corresponding to a scanning task may be one.

[0053] In this embodiment, after the scanning task is acquired, the scanning task is parsed, so as to determine the storage bucket to be scanned corresponding to the scanning task, ie, the storage bucket to be scanned, in the storage system.

[0054] Step S120 : dividing the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and splitting the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division.

[0055] Buckets are used to store objects, and therefore have corresponding storage areas.

[0056] In order to improve the execution speed and success rate of the scanning task, in this embodiment, the storage area of the storage bucket to be scanned is divided into multiple sub-areas to be scanned, and then the scanning task is split based on the sub-areas to be scanned to obtain multiple scanning sub-tasks, each scanning sub-task corresponds to a sub-area to be scanned, thereby realizing the splitting of the scanning task into multiple scanning sub-tasks of different sub-areas to be scanned.

[0057] Among them, the specific method of dividing the storage bucket to be scanned into multiple sub-areas to be scanned can be flexibly set according to actual needs. For example, if the storage system is a distributed storage system, the areas in the storage bucket to be scanned that belong to the same storage node can be divided into the same sub-area to be scanned, and multiple sub-areas to be scanned can be obtained to realize the execution of scanning subtasks in units of storage nodes; or, the storage bucket to be scanned can be divided into multiple sub-areas to be scanned of preset sizes; of course, other division methods can also be used, and they are not limited in this embodiment.

[0058] Step S130 : executing the split scanning subtasks to search for target objects in the corresponding sub-areas to be scanned.

[0059] The target object is the object that needs to be queried from the bucket, and can be configured by the user according to actual needs.

[0060] After the scanning task is split, the resulting scanning subtasks can be executed. During the execution of the scanning subtasks, the sub-area to be scanned corresponding to the scanning subtasks is scanned to search for target objects in the sub-area to be scanned.

[0061] It should be noted that in order to improve the execution rate, during the execution of multiple scanning subtasks corresponding to the same scanning task, the scanning subtasks can be executed in parallel, thereby realizing parallel scanning within the storage bucket and further improving the scanning speed.

[0062] In this embodiment, the scanning task is parsed to determine a storage bucket to be scanned corresponding to the scanning task in the storage system; wherein the storage bucket includes a carrier for object storage; the storage area of the storage bucket to be scanned is divided into multiple sub-areas to be scanned, and the scanning task is split into multiple scanning sub-tasks based on the multiple sub-areas to be scanned obtained by the division; the scanning sub-tasks obtained by the split are executed to query the target object from the corresponding sub-area to be scanned. In this way, on the one hand, the scanning task is split into multiple scanning sub-tasks, and each scanning sub-task is executed, thereby improving the scanning speed and success rate; on the other hand, the storage area of the storage bucket to be scanned is divided into multiple sub-areas to be scanned, and the corresponding sub-areas to be scanned are scanned when the scanning sub-task is executed, which can avoid repeated scanning and can realize parallel scanning within the storage bucket, further improving the scanning speed.

[0063] In an exemplary embodiment, a storage system includes multiple storage tables for storing data. A single storage table can be deployed on a storage node, and a storage node can host one or more storage tables. The capacities of different storage tables can be the same, and the capacity of a storage table can be flexibly adjusted based on actual needs. Data within a storage bucket is stored in a storage table. The number of storage tables a bucket contains is determined based on the amount of data stored in the bucket. If the amount of data stored in a bucket is small, multiple storage buckets can share a single storage table. If the amount of data stored in a bucket is large, a single storage bucket can occupy multiple storage tables.

[0064] In order to facilitate scanning, if the storage system includes multiple storage tables, Figure 2 In step S120 of the illustrated embodiment, the process of dividing the storage area of the storage bucket to be scanned into multiple sub-areas to be scanned may include: dividing the areas in the storage area of the storage bucket to be scanned belonging to the same storage table into the same sub-area to be scanned, thereby obtaining multiple sub-areas to be scanned.

[0065] In some embodiments, the process of dividing the areas in the storage area of the storage bucket to be scanned that belong to the same storage table into the same sub-area to be scanned to obtain multiple sub-areas to be scanned may include: determining the index range corresponding to the storage bucket to be scanned, and if the index range includes multiple index sub-tables, dividing the storage area corresponding to the same index sub-table into the same sub-area to be scanned to obtain multiple sub-areas to be scanned, thereby facilitating scanning the corresponding sub-areas to be scanned based on the index sub-table, thereby further improving the scanning speed.

[0066] It should be noted that in order to speed up data retrieval and optimize the performance of the storage system, the storage system may include a global index table for indexing the data in the storage area contained in the storage system. The global index table may include multiple index sub-tables, one index sub-table corresponding to one storage table, and the index sub-table is used to index the data in the corresponding storage table. The global index table is sorted according to the index key value and has the characteristics of order between tables and order within tables. For example, see Figure 3 As shown, it includes index sub-table 0, index sub-table 1 and index sub-table N (N is an integer greater than or equal to 2). The index key values of index sub-table 0, index sub-table 1 and index sub-table N have the characteristics of being ordered between tables and ordered within tables.

[0067] Under the condition that the storage system includes a global index table, and the global index table includes multiple index sub-tables, in order to improve the scanning speed, in this embodiment, the index range corresponding to the storage bucket to be scanned can be first determined. If there are multiple index sub-tables in the index range, the storage area corresponding to each index sub-table in the multiple index sub-tables is used as a sub-area to be scanned, thereby obtaining multiple sub-areas to be scanned, that is, one sub-area to be scanned corresponds to the storage table of one index sub-table. For example, see Figure 4 As shown, assuming that the index range of the bucket to be scanned is Figure 3 As shown in az, the index range is split based on the index sub-table, thereby obtaining scanning subtasks with (a,e], (e,j], (j,…], and (p,z] as the sub-areas to be scanned. In this way, the scanning task can be split based on the index information. When the scanning subtask is subsequently executed, the target object can be queried from the sub-area to be scanned based on the index data in the index sub-table corresponding to the sub-area to be scanned, thereby improving the retrieval speed.

[0068] See also Figure 5 , Figure 5 for Figure 2 The flowchart of step S130 in the embodiment shown is in an exemplary embodiment. Figure 5 As shown, the process of executing the split scanning subtasks may include steps S210 to S240, which are described in detail as follows:

[0069] Step S210 , sequentially scanning the sub-areas to be scanned corresponding to the scanning sub-tasks, and recording the scanning positions of the scanning sub-tasks and the first scanning results corresponding to the scanned areas at every preset period during the scanning process.

[0070] The preset period is the time interval for recording the scanning position and the scanning result, and its value can be flexibly set according to actual needs, for example, it can be 1 second, 2 seconds, etc.

[0071] The scanning position of the scanning subtask is the scanning position corresponding to the scanning subtask during the execution of the scanning subtask. The first scanning result is the scanning result corresponding to the area scanned by the scanning subtask when the preset period is reached during the execution of the scanning subtask.

[0072] During the execution of a scanning subtask, an exception may occur, requiring the scanning subtask to be executed again. To avoid rescanning a previously scanned area during the re-execution process, resulting in a slow scanning rate, in this embodiment, for any scanning subtask, the sub-area to be scanned corresponding to the scanning subtask is scanned sequentially. During the scanning process, the scanning position of the scanning subtask and the first scanning result corresponding to the scanned area are recorded at every preset period. That is, after the preset period arrives, the scanning position corresponding to the scanning subtask at that moment and the scanning result corresponding to the scanned area are recorded. For example, assuming the preset period is 3 seconds, the scanning position of scanning subtask a1 and the scanning result corresponding to the storage area scanned before that moment (i.e., the first scanning result) are recorded at 12:05:20. When the next period arrives, i.e., 12:05:23, the scanning position of scanning subtask a1 and the scanning result corresponding to the storage area scanned before that moment are recorded again.

[0073] In one example, to reduce storage pressure, the record may only store the latest scanning position and the latest first scanning result of the scanning subtask; of course, in another example, the record may also store the scanning position and the first scanning result of the scanning subtask in different cycles.

[0074] Step S220: If an exception occurs in the scanning subtask, the latest scanning position and the latest first scanning result of the scanning subtask are obtained from the record.

[0075] The latest first scanning result is the scanning result corresponding to the area scanned before the latest scanning position.

[0076] If an exception occurs in a scanning subtask, in order to avoid repeated scanning, the latest scanning position and the latest first scanning result of the scanning subtask are obtained from the record.

[0077] Step S230 , starting from the latest scanning position, scanning the unscanned area in the corresponding sub-area to be scanned to obtain a second scanning result.

[0078] After obtaining the latest scanning position and the latest first scanning result of the scanning subtask, the latest scanning position is used as the starting point to scan the unscanned area in the sub-area to be scanned corresponding to the scanning subtask, thereby obtaining the second scanning result, realizing "breakpoint resumption", avoiding repeated scanning, and improving the scanning speed.

[0079] Step S240: Merge the latest first scanning result and the second scanning result to obtain a scanning result of the scanning subtask.

[0080] The latest first scanning result and the second scanning result of the scanning subtask are merged to obtain the scanning result of the scanning subtask. For example, in one example, assuming that the area to be scanned corresponding to the scanning subtask B1 is b1-b9, during the execution of the scanning subtask B1, b1 is used as the starting point and b9 as the end point, and b1-b9 are scanned in sequence. At time t1, the scanning position of the scanning subtask B1 is b2, and the first scanning result is c2; at time t1+T (where T is a preset period), the scanning position of the scanning subtask B1 is b5, and the first scanning result is c5; between time t1+T and time t1+2*T, an abnormality occurs in the scanning subtask B1. At this time, the latest scanning position obtained is b5, and the latest first scanning result is c5. Therefore, starting from b5, the unscanned area in b1-b9, i.e., b6-b9, is scanned to obtain the second scanning result c6. Since c5 is the scanning result of b1-b5 and c6 is the scanning result of b6-b9, c5 and c6 are merged to obtain the scanning result of the scanning subtask B1.

[0081] In some implementations, after obtaining the scan result of a scan subtask, in order to reduce storage pressure, the scan position and the first scan result of the scan subtask may be deleted from the record.

[0082] In this embodiment, the sub-areas to be scanned corresponding to the scanning subtasks are scanned in sequence, and the scanning position of the scanning subtask and the first scanning result corresponding to the scanned storage area are recorded at preset intervals during the scanning process; if an abnormality occurs in the scanning subtask, the latest scanning position and the latest first scanning result of the scanning subtask are obtained from the record, and the unscanned storage area in the corresponding sub-area to be scanned is scanned with the latest scanning position as the starting point to obtain the second scanning result; the latest first scanning result and the second scanning result are merged to obtain the scanning result of the scanning subtask. In this way, if an abnormality occurs during the execution of the scanning subtask, the scan is started from the point where the abnormality occurs when it is executed again, thereby avoiding repeated scanning, thereby improving the scanning speed and improving the task execution efficiency.

[0083] See also Figure 6 , Figure 6 In the case where the number of scanning tasks includes multiple, Figure 2 The flowchart of step S130 in the embodiment shown is in an exemplary embodiment. Figure 6 As shown, the process of executing the split scanning subtask may include steps S310 to S320, which are described in detail as follows:

[0084] Step S310 : obtaining the total number of scanning subtasks being executed from the scanning subtasks corresponding to the plurality of scanning tasks.

[0085] There can be multiple scan tasks that need to be executed at the same time. Under this condition, the total number of scan subtasks currently being executed among the scan subtasks corresponding to these multiple scan tasks can be obtained. For example, suppose that at the same time, the scan tasks that need to be executed include D1, D2, and D3. Among the multiple scan subtasks corresponding to D1, 2 scan subtasks are currently being executed, among the multiple scan subtasks corresponding to D2, 3 scan subtasks are currently being executed, and among the multiple scan subtasks corresponding to D3, 5 scan subtasks are currently being executed. The total number is 2 + 3 + 5 = 10.

[0086] Step S320 : If the total number obtained is less than the first threshold, unexecuted scan subtasks are extracted from the scan subtasks corresponding to the multiple scan tasks, and the extracted scan subtasks are executed.

[0087] To prevent scan tasks from occupying a large amount of resources and impacting other storage system functions, a first threshold is set. The first threshold is the maximum number of scan subtasks that can be executed in parallel within multiple scan tasks. The specific value of the first threshold can be flexibly set based on actual needs. For example, in one example, the first threshold can be a preset value, such as 3, 5, or 8. Alternatively, in another example, the first threshold can be flexibly adjusted based on the storage system load. If the storage system load is high, the first threshold can be lowered; if the storage system load is low, the first threshold can be increased.

[0088] In this embodiment, after obtaining the total number of executing scan subtasks in multiple scan tasks, if the total number is less than a first threshold, unexecuted scan subtasks are extracted from the scan subtasks corresponding to the multiple scan tasks and executed. If the total number is greater than or equal to the first threshold, the system waits until the total number is less than the first threshold before extracting and executing the scan subtasks.

[0089] The specific method for extracting unexecuted scan subtasks from the scan subtasks corresponding to the multiple scan tasks can be flexibly configured based on actual needs. For example, in one embodiment, a target number of scan subtasks can be first determined, and then the target number of scan subtasks can be extracted from the scan subtasks corresponding to the multiple scan tasks, and then the extracted scan subtasks can be executed, where the target number of scan subtasks is the difference between the first threshold and the total number of scan subtasks.

[0090] The specific method of executing the extracted scanning subtask can be flexibly set according to actual needs. For example, the extracted scanning subtask can be executed according to the method of steps S210 to S240.

[0091] In this embodiment, the total number of scanning subtasks being executed is obtained from the scanning subtasks corresponding to multiple scanning tasks. If the total number obtained is less than a first threshold, the unexecuted scanning subtasks are extracted from the scanning subtasks corresponding to the multiple scanning tasks, and the extracted scanning subtasks are executed, thereby avoiding the situation where the scanning tasks occupy a large amount of resources and affect other functions of the storage system.

[0092] See also Figure 7 , Figure 7 for Figure 6 The flowchart of step S320 in the embodiment shown is in an exemplary embodiment. Figure 7 As shown, the process of extracting unexecuted scan subtasks from the scan subtasks corresponding to the multiple scan tasks may include steps S321 to S323, which are described in detail as follows:

[0093] Step S321 : obtaining the arrangement order of the plurality of scanning tasks, and determining the first scanning task corresponding to the most recently extracted scanning subtask.

[0094] In this embodiment, multiple scanning tasks can be sorted to obtain an arrangement order of the multiple scanning tasks. The sorting method can be flexibly set according to actual needs. For example, the scanning tasks can be sorted according to the triggering time of the scanning tasks. Of course, other sorting methods can also be used.

[0095] The first scanning task is the scanning task to which the most recently extracted scanning subtask belongs.

[0096] In this embodiment, the arrangement order of multiple scanning tasks is obtained, and the first scanning task is determined.

[0097] Step S322 : selecting the scanning task that is one position behind the first scanning task according to the arrangement order as the second scanning task.

[0098] After obtaining the arrangement order and determining the first scanning task, a scanning task whose arrangement position is one position after the first scanning task is determined, and the determined scanning task is used as the second scanning task.

[0099] In some embodiments, if the first scanning subtask is ranked last, in order to avoid failure to extract the scanning subtask, the scanning subtask ranked first may be used as the second scanning subtask, thereby achieving cyclic extraction.

[0100] Step S323: extract an unexecuted scanning subtask from the scanning subtasks corresponding to the second scanning task.

[0101] After determining the second scanning task, an unexecuted scanning subtask is extracted from the multiple scanning subtasks corresponding to the second scanning task and executed. For example, in one example, assuming there are three scanning tasks arranged in the order of D1, D2, and D3, after extracting and executing a scanning subtask from the multiple scanning subtasks corresponding to D2, if the total number of scanning subtasks being executed among the three scanning tasks is less than a first threshold, then the first scanning subtask is D2 and the second scanning subtask is D3. An unexecuted scanning subtask is extracted from the multiple scanning subtasks corresponding to D3 and executed.

[0102] After extracting an unexecuted scanning subtask from the scanning subtask corresponding to the second scanning task, you can return to step S310, so as to extract the unexecuted scanning subtasks from multiple scanning tasks in sequence according to the arrangement order, avoiding the situation where there are a large number of scanning subtasks being executed in a certain scanning task and other scanning tasks cannot be executed, thereby achieving balanced scheduling.

[0103] In this embodiment, scanning subtasks are extracted from multiple scanning subtasks in sequence according to the arrangement order, thereby avoiding the situation where too many scanning subtasks are being executed in a certain scanning task, occupying a large amount of resources, and causing slow execution progress of other scanning tasks, thereby achieving balanced scheduling.

[0104] In an exemplary embodiment, in order to prevent large scanning tasks from occupying resources for a long time and to ensure fair scheduling of small scanning tasks, step S323 may include steps 1.1 to 1.3, which are described in detail as follows:

[0105] Step 1.1: Obtain the number of scanning subtasks being executed from the scanning subtasks corresponding to the second scanning task.

[0106] In this embodiment, the number of scanning subtasks being executed among the multiple scanning subtasks corresponding to the second scanning task is obtained.

[0107] Step 1.2: If the number of scan subtasks being executed is less than the second threshold, extract an unexecuted scan subtask from the scan subtasks corresponding to the second scan task.

[0108] To prevent a situation where too many scanning tasks are currently executing within a single scanning task, slowing down the progress of other scanning tasks, a second threshold is set for each scanning task. This threshold represents the maximum number of scanning subtasks that can be executed in parallel within the multiple scanning subtasks corresponding to a single scanning task. The specific value of this second threshold can be flexibly set based on actual needs. Different scanning tasks can have different corresponding second thresholds. Alternatively, to reduce processing resources, different scanning tasks can have the same second threshold.

[0109] In this embodiment, if the number of scanning subtasks being executed among the multiple scanning subtasks corresponding to the second scanning task is less than the second threshold, an unexecuted scanning subtask is extracted from the multiple scanning subtasks corresponding to the second scanning task. After the extraction, the process may return to step S310.

[0110] Step 1.3: If the number of scan subtasks being executed is greater than or equal to the second threshold, extract an unexecuted scan subtask from the scan tasks whose sorting position is after the second scan task.

[0111] If the number of scan subtasks being executed is greater than or equal to the second threshold, an unexecuted scan subtask is extracted from the scan tasks that are sorted after the second scan task. The specific method for extracting an unexecuted scan subtask from the scan tasks that are sorted after the second scan task is similar to the method for extracting an unexecuted scan subtask from the multiple scan subtasks corresponding to the second scan task, and is not further described here.

[0112] In this embodiment, through the aforementioned steps 1.1 to 1.3, it is possible to avoid a situation where a certain scanning task occupies a large amount of resources for a long time, thereby achieving balanced scheduling.

[0113] In an exemplary embodiment, under the condition that the scanning task includes at least one of a pre-stored scanning task and a real-time scanning task, Figure 2 In step S130 of the illustrated embodiment, the process of executing the split scanning subtasks may include: if the scanning subtasks to be executed include scanning subtasks corresponding to pre-stored scanning tasks and scanning subtasks corresponding to real-time scanning tasks, the scanning subtasks corresponding to real-time scanning tasks are executed first.

[0114] It should be noted that the pre-stored scanning task is a pre-stored scanning task. For example, it can be a periodic scanning task, that is, a scanning task that is executed once every certain period. The period can be flexibly set according to actual needs. For example, it can be a scanning task executed once a day, a scanning task executed once a week, a scanning task executed once a month, etc.

[0115] A real-time scanning task is a scanning task that is received in real time and needs to be executed in real time. For example, it can be a temporary scanning task, a one-time scanning task (ie, a scanning task that is executed only once), and the like.

[0116] If at the same time, the scan subtasks waiting to be executed include the scan subtasks corresponding to the pre-stored scan task and the scan subtasks corresponding to the real-time scan task, the scan subtasks corresponding to the real-time scan task can be executed first, so that the real-time scan task can be executed in time.

[0117] Among them, the method of prioritizing the execution of the scanning subtasks corresponding to the real-time scanning task can be flexibly set according to actual needs. For example, the scanning subtasks corresponding to the pre-stored scanning task can be executed after the scanning subtasks corresponding to the real-time scanning task are completed; or, the second threshold corresponding to the real-time scanning task can be greater than the second threshold corresponding to the pre-stored scanning task, so that at the same time, the number of scanning subtasks being executed in the real-time scanning task is greater than the number of scanning subtasks being executed in the pre-stored scanning task, thereby enabling the real-time scanning task to be completed in a timely manner.

[0118] In this embodiment, if the scanning subtasks to be executed include scanning subtasks corresponding to pre-stored scanning tasks and scanning subtasks corresponding to real-time scanning tasks, the scanning subtasks corresponding to the real-time scanning tasks are executed first. In this way, during the execution of pre-stored scanning tasks, if a real-time scanning task is received, the real-time scanning task can be processed in a timely manner, thereby reducing the time from creation to completion of the real-time scanning task, improving the real-time performance of the real-time scanning task, reducing the user's waiting time, and improving the user experience.

[0119] In an exemplary embodiment, the data scanning method may further include: during the execution of the split scanning subtasks, if the number of execution failures of any scanning subtask is less than a preset number threshold, the scanning subtask may be re-executed; if the number of execution failures of the scanning subtask is equal to the preset number threshold, in order to improve the execution success rate, the scanning subtask may be executed again after waiting for a preset time; or, if the number of execution failures of the scanning subtask is equal to the preset number threshold, in order to reduce resource utilization, the execution of the scanning subtask may be terminated and the execution result of the scanning subtask may be output as execution failure. The preset number threshold can be flexibly set according to actual needs, for example, it can be set to 3, 4, 5, etc.; the preset time can be flexibly set according to actual needs, for example, 50 seconds, 100 seconds, etc.

[0120] In an exemplary embodiment, Figure 2 After step S130 in the illustrated embodiment, the data scanning method may further include: after the target scanning subtask among the multiple scanning subtasks is executed, obtaining the scanning result corresponding to the target scanning subtask; outputting the scanning result corresponding to the target scanning subtask, and processing the scanning result corresponding to the target scanning subtask.

[0121] The target scanning subtask is any completed scanning subtask among the multiple scanning subtasks corresponding to the scanning task. After any scanning subtask corresponding to the scanning task is completed, the scanning result corresponding to the scanning subtask can be obtained, output, and processed. This allows the scanning result of any scanning subtask to be processed as soon as one of the scanning subtasks is completed, without having to wait for the entire scanning task to be completed. This enables rolling delivery and improves response speed.

[0122] The processing method for the scan results corresponding to the target scan subtask can be flexibly configured based on actual needs. In one example, downstream services can be notified to obtain and process the scan results of the target scan subtask. Downstream services include, but are not limited to, lifecycle management services, intelligent tiered storage services, cross-region replication services, inspection services, and reconciliation services. The lifecycle management service automatically performs certain actions on objects that meet certain rules under specified conditions, including, but not limited to, expiration deletion and demotion. In one example, log data can be automatically archived after 30 days or automatically deleted after 2 years. The cross-region replication service replicates objects from one availability zone to another. The intelligent tiered storage service provides a hot and cold tiering mechanism for data, automatically switching between hot and cold tiers based on user data access patterns, thereby reducing user data storage costs. The inspection service is used to test the storage system and identify issues and potential risks. The reconciliation service is used to cross-check various bills to verify that resource consumption is consistent with actual account expenditures.

[0123] In an exemplary embodiment, Figure 2 After step S130 in the illustrated embodiment, the data scanning method may further include: after the multiple scanning subtasks corresponding to the scanning task are all executed, merging the scanning results corresponding to the multiple scanning subtasks to obtain the scanning result corresponding to the scanning task.

[0124] The following describes a specific application scenario of the embodiment of the present application in detail. In this embodiment, the scanning task is taken as an example to illustrate, wherein the inventory task corresponds to a bucket to be scanned. Figure 8As shown, the data scanning system includes a data scanning device, a database, an ordered storage service, and downstream services. The data scanning device is implemented as an inventory node cluster for implementing inventory tasks. The inventory node cluster may include a scheduler, multiple working nodes, and an index parsing node. The ordered storage service may be Lavadb (a distributed ordered storage service for object indexes of an object storage service). Downstream services include an external product inventory service (for delivering the inventory output by the inventory task to the outside), a lifecycle service, a cross-region replication service, and an inspection service.

[0125] Among them, the storage system can store the object metadata in the database and obtain the task information of the inventory task to create the inventory task based on the task information and write the inventory task into the database. The ordered storage service can generate a global index table based on the written object index. The scheduler in the inventory cluster can obtain the inventory task and parse the index range of the inventory task based on the global index table, split the inventory task according to the index sub-table contained in the index range, thereby obtaining multiple sub-tasks, wherein the sub-area to be scanned corresponding to each sub-task is the storage area corresponding to an index sub-table. The scheduler can concurrently schedule sub-tasks to the working node, and the working node executes the sub-tasks scheduled by the scheduler and obtains the execution results, wherein a working node can execute multiple sub-tasks in parallel, thereby realizing concurrent execution of sub-tasks and improving execution speed. When the multiple sub-tasks corresponding to the inventory task are completed, the scheduler can summarize the execution results of the multiple sub-tasks, and can also perform operations such as recycling task resources.

[0126] To improve fault tolerance, during the execution of a subtask, if the number of failures of a subtask is less than three, the scheduler can reschedule the subtask to the working node so that the subtask can be executed again. To avoid the situation where the subtask fails and the progress that has already been executed is re-executed, thereby wasting resources, the working node can record the execution progress of the subtask (including the scan position and the first scan result) at preset intervals during the execution of the subtask. When an exception occurs in the subtask, the working node obtains the latest execution progress from the record and executes the unexecuted part of the subtask based on the latest execution progress.

[0127] In order to avoid long-term resource occupation by large list tasks and to ensure fair scheduling of short list tasks, the scheduler schedules subtasks corresponding to multiple list tasks. Figure 9As shown, the list tasks can be sorted first. When the cursor moves to a certain list task, the total number of subtasks being executed in multiple list tasks is obtained. If the total number does not exceed the first threshold, an unexecuted subtask is scheduled from the list task where the cursor is located to the working node, and the cursor is moved to the next list task so that the next scheduling will not be concentrated on the same list task; if the total number exceeds the first threshold, the scheduling is suspended.

[0128] In the process of scheduling an unexecuted subtask from the list task where the cursor is located to the working node, the number of subtasks being executed in the list task can be obtained first. If the number is less than the second threshold, an unexecuted subtask is scheduled from the list task to the working node; if the number is equal to the second threshold, the cursor is moved to the next list task to find the subtask to be scheduled. For example, Figure 9 In the figure, the second threshold corresponding to list task 1, list task 2, and list task x (x is an integer greater than or equal to 2) is 8, where the squares filled with slashes represent subtasks being executed, and the blank squares represent subtasks that can be scheduled for parallel execution. The list task where the cursor is located is list task 2, and the number of subtasks being executed in list task 2 is 2, which is less than the second threshold of 8. Therefore, unexecuted subtasks can be scheduled from the multiple subtasks corresponding to list task 2; if the cursor moves to list task x, since the number of subtasks being executed in list task x is 8, it has reached the second threshold. Therefore, subtasks can no longer be scheduled from the multiple subtasks corresponding to list task x.

[0129] After completing a subtask and obtaining the CSV file for the subtask, the working node can notify downstream services, such as the external product inventory service, lifecycle service, cross-region replication service, and inspection service, so that the downstream services can process the CSV file of the subtask. This allows the corresponding CSV file to be processed after completing a subtask, without having to wait for the entire inventory task to be completed before processing, thereby speeding up the processing process.

[0130] Inventory tasks include periodic and real-time tasks. If a real-time task is received during the execution of a periodic task, it can be stored in a preset directory, such as the LavaDB directory / meta / temporary / . The scheduler can periodically poll this directory to retrieve real-time tasks and prioritize the subtasks corresponding to real-time tasks to ensure fast execution of the real-time task and improve the real-time performance of temporary tasks.

[0131] The solution in this embodiment can improve the execution speed of the list tasks.

[0132] See also Figure 10 , Figure 10 FIG. 1 is a block diagram of a data scanning device according to an exemplary embodiment of the present invention. Figure 10 As shown, the device includes:

[0133] The parsing module 1001 is configured to parse the scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in the storage system; wherein the storage bucket includes a carrier of object storage.

[0134] The splitting module 1002 is configured to divide the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and split the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division.

[0135] The execution module 1003 is configured to execute the split scanning subtasks to search for target objects in the corresponding sub-areas to be scanned.

[0136] In another exemplary embodiment, when the number of scanning tasks includes a plurality, the executing module 1003 includes:

[0137] The total number obtaining module is configured to obtain the total number of scanning subtasks being executed from the scanning subtasks corresponding to the multiple scanning tasks.

[0138] The task execution module is configured to extract unexecuted scanning subtasks from the scanning subtasks corresponding to the plurality of scanning tasks if the acquired total number is less than a first threshold, and execute the extracted scanning subtasks.

[0139] In another exemplary embodiment, the task execution module includes:

[0140] The first determining module is configured to obtain an arrangement order of the plurality of scanning tasks and determine a first scanning task corresponding to the most recently extracted scanning subtask.

[0141] The second determining module is configured to select the scanning task that is one position after the first scanning task according to the arrangement order as the second scanning task.

[0142] The extraction module is configured to extract an unexecuted scanning subtask from the scanning subtasks corresponding to the second scanning task.

[0143] In another exemplary embodiment, the extraction module includes:

[0144] The quantity acquisition module is configured to acquire the quantity of the scanning subtasks being executed from the scanning subtasks corresponding to the second scanning task.

[0145] The first task extraction module is configured to extract an unexecuted scanning subtask from the scanning subtasks corresponding to the second scanning task if the number of the acquired scanning subtasks being executed is less than a second threshold.

[0146] The second task extraction module is configured to extract an unexecuted scanning subtask from the scanning tasks that are sorted after the second scanning task if the number of the acquired scanning subtasks being executed is greater than or equal to a second threshold.

[0147] In another exemplary embodiment, the apparatus further comprises:

[0148] The result acquisition module is configured to acquire the scanning result corresponding to the target scanning subtask after the target scanning subtask among the multiple scanning subtasks is completed.

[0149] The processing module is configured to output the scanning result corresponding to the target scanning subtask and process the scanning result corresponding to the target scanning subtask.

[0150] In another exemplary embodiment, under the condition that the scanning task includes at least one of a pre-stored scanning task and a real-time scanning task, the executing module 1003 includes:

[0151] The subtask execution module is configured to, if the scan subtasks to be executed include scan subtasks corresponding to pre-stored scan tasks and scan subtasks corresponding to real-time scan tasks, give priority to executing the scan subtasks corresponding to the real-time scan tasks.

[0152] In another exemplary embodiment, the execution module 1003 includes:

[0153] The recording module is configured to sequentially scan the to-be-scanned sub-areas corresponding to the scanning sub-tasks, and record the scanning positions of the scanning sub-tasks and the first scanning results corresponding to the scanned areas at every preset period during the scanning process.

[0154] The record acquisition module is configured to acquire the latest scanning position and the latest first scanning result of the scanning subtask from the record if an exception occurs in the scanning subtask.

[0155] The scanning module is configured to scan an unscanned area in the sub-area to be scanned, starting from the latest scanning position, to obtain a second scanning result.

[0156] The merging module is configured to merge the latest first scanning result and the second scanning result to obtain the scanning result of the scanning subtask.

[0157] In another exemplary embodiment, when the storage system includes multiple storage tables, the splitting module 1002 includes:

[0158] The region splitting module is configured to divide the regions belonging to the same storage table in the storage region of the to-be-scanned storage bucket into the same to-be-scanned sub-region to obtain multiple to-be-scanned sub-regions.

[0159] It should be noted that the data scanning device provided in the above embodiment and the data scanning method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0160] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by one or more processors, enables the electronic device to implement the data scanning method provided in the above-mentioned embodiments.

[0161] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0162] It should be noted that Figure 11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0163] like Figure 11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1102 or the program loaded from the storage part 1108 into the random access memory (RAM) 1103, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1103. The CPU 1101, ROM 1102 and RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0164] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output section 1107 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1110 as needed, so that computer programs read from the removable media can be installed in the storage section 1108 as needed.

[0165] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the various functions defined in the system of the present application are executed.

[0166] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0169] Another aspect of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of an electronic device, the electronic device implements the aforementioned method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0170] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions that, when executed by a processor, implement the methods provided in the above embodiments. The computer instructions may be stored in a computer-readable storage medium; a processor of an electronic device may read the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, causing the electronic device to perform the methods provided in the above embodiments.

[0171] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data scanning method, characterized in that: The method comprises: Parsing the scan task to determine a to-be-scanned storage bucket corresponding to the scan task in a storage system; wherein the storage bucket includes a carrier for object storage; Dividing the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and splitting the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division; Execute the split scanning subtasks to query the target object from the corresponding sub-area to be scanned; The scanning subtasks obtained by executing the splitting include: Obtain the total number of scanning subtasks being executed from the scanning subtasks corresponding to the multiple scanning tasks; If the total number obtained is less than the first threshold, the unexecuted scanning subtasks are extracted from the scanning subtasks corresponding to the multiple scanning tasks, and the extracted scanning subtasks are executed; wherein, if the number of scanning subtasks being executed in the second scanning task is less than the second threshold, the unexecuted scanning subtasks are extracted from the second scanning task; if the number of scanning subtasks being executed in the second scanning task is greater than or equal to the second threshold, the unexecuted scanning subtasks are extracted from the scanning task whose sorting position is after the second scanning task; the second scanning task is a scanning task whose sorting position is one place after the first scanning task, and the first scanning task is the scanning task corresponding to the most recently extracted scanning subtask.

2. The method according to claim 1, wherein The extracting unexecuted scanning subtasks from the scanning subtasks corresponding to the plurality of scanning tasks includes: Obtaining an arrangement order of the plurality of scanning tasks, and determining a first scanning task corresponding to the most recently extracted scanning subtask; According to the arrangement order, the scanning task whose sorting position is one position after the first scanning task is used as the second scanning task; An unexecuted scanning subtask is extracted from the scanning subtasks corresponding to the second scanning task.

3. The method according to claim 2, wherein The extracting an unexecuted scanning subtask from the scanning subtasks corresponding to the second scanning task includes: Obtaining the number of scanning subtasks being executed from the scanning subtasks corresponding to the second scanning task; If the number of scan subtasks being executed is less than a second threshold, extracting an unexecuted scan subtask from the scan subtasks corresponding to the second scan task; If the number of scan subtasks being executed is obtained to be greater than or equal to a second threshold, an unexecuted scan subtask is extracted from the scan tasks whose sorting position is after the second scan task.

4. The method according to claim 1, wherein After executing the split scanning subtasks, the method further includes: After the target scanning subtask among the multiple scanning subtasks is executed, obtaining a scanning result corresponding to the target scanning subtask; Output the scanning result corresponding to the target scanning subtask, and process the scanning result corresponding to the target scanning subtask.

5. The method according to claim 1, wherein The scanning task includes at least one of a pre-stored scanning task and a real-time scanning task, and the scanning subtasks obtained by executing the splitting include: If the scan subtasks to be executed include the scan subtask corresponding to the pre-stored scan task and the scan subtask corresponding to the real-time scan task, the scan subtask corresponding to the real-time scan task is executed first.

6. The method according to claim 1, wherein The scanning subtasks obtained by executing the splitting include: sequentially scanning the to-be-scanned sub-areas corresponding to the scanning sub-tasks, and recording the scanning positions of the scanning sub-tasks and the first scanning results corresponding to the scanned areas at predetermined intervals during the scanning process; If an exception occurs in the scanning subtask, the latest scanning position and the latest first scanning result of the scanning subtask are obtained from the record; Taking the latest scanning position as a starting point, scanning an unscanned area in the corresponding sub-area to be scanned to obtain a second scanning result; The latest first scanning result and the second scanning result are merged to obtain a scanning result of the scanning subtask.

7. The method according to claim 1, wherein The storage system includes a plurality of storage tables; the storage area of the to-be-scanned storage bucket is divided into a plurality of to-be-scanned sub-areas, including: The areas belonging to the same storage table in the storage area of the to-be-scanned storage bucket are divided into the same to-be-scanned sub-area to obtain a plurality of to-be-scanned sub-areas.

8. A data scanning device, characterized in that: The device comprises: A parsing module configured to parse the scanning task to determine a to-be-scanned storage bucket corresponding to the scanning task in the storage system; wherein the storage bucket includes a carrier of object storage; a splitting module configured to divide the storage area of the to-be-scanned storage bucket into a plurality of to-be-scanned sub-areas, and to split the scanning task into a plurality of scanning sub-tasks according to the plurality of to-be-scanned sub-areas obtained by the division; An execution module configured to execute the split scanning subtasks to search for target objects in the corresponding sub-areas to be scanned; The scanning subtasks obtained by executing the splitting include: Obtain the total number of scanning subtasks being executed from the scanning subtasks corresponding to the multiple scanning tasks; If the total number obtained is less than the first threshold, the unexecuted scanning subtasks are extracted from the scanning subtasks corresponding to the multiple scanning tasks, and the extracted scanning subtasks are executed; wherein, if the number of scanning subtasks being executed in the second scanning task is less than the second threshold, the unexecuted scanning subtasks are extracted from the second scanning task; if the number of scanning subtasks being executed in the second scanning task is greater than or equal to the second threshold, the unexecuted scanning subtasks are extracted from the scanning task whose sorting position is after the second scanning task; the second scanning task is a scanning task whose sorting position is one place after the first scanning task, and the first scanning task is the scanning task corresponding to the most recently extracted scanning subtask.

9. An electronic device, characterized in that: include: one or more processors; A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the data scanning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the data scanning method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the data scanning method according to any one of claims 1 to 7 is implemented.

Images