Task data scanning method and device based on byte order fragmentation, equipment and medium
By using byte order sharding expansion and subtask mapping, combined with load balancing strategies, real-time monitoring, and dynamic adjustments, the problem of low task data scanning efficiency was solved, achieving efficient and reliable data scanning and merging, and improving resource utilization and fault handling capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-06
- Publication Date
- 2026-04-14
AI Technical Summary
Existing technologies suffer from several drawbacks in task data scanning, including insufficient concurrent scheduling mechanisms and multi-node resource adaptability, lack of dynamic perception and automatic adjustment capabilities in load allocation, failure to achieve fine-grained breakpoint resumption of scanning in fault tolerance and recovery mechanisms, and reliance on manual intervention and lack of flexibility in sharding strategies.
A task data scanning method based on byte order sharding is adopted. The initial prefix of the target scanning task is obtained and expanded by byte order sharding to generate a set of sub-prefixes. The sub-prefixes are mapped to sub-tasks and assigned to preset working nodes to generate task scanning requests. The task execution progress is monitored in real time, and the load balancing strategy is dynamically adapted to realize the merging and continued scanning of task data.
It significantly improves the efficiency of task data scanning, reduces the time spent enumerating massive amounts of data, dynamically adapts sharding logic to avoid resource waste, ensures data integrity and accuracy, and improves cluster resource utilization and fault handling efficiency.
Smart Images
Figure CN121858243A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for scanning task data based on byte order fragmentation. Background Technology
[0002] In a distributed key-value storage system, byte order sharding refers to logically dividing the entire key space into multiple continuous and non-overlapping sub-ranges based on the byte sequence order of the key. Each sub-range can be scanned independently. Task data scanning is achieved by splitting the data traversal task and distributing it to multiple nodes for parallel execution.
[0003] In the fintech field, the management platforms of medium and large banks face millions of IT assets distributed across tens of thousands of branches at home and abroad. Traditional centralized scanning takes more than a month to complete a full asset screening. However, by adopting a cross-regional distributed scanning architecture, the head office's centralized management platform coordinates the configuration strategy and deploys more than 500 lightweight scanning engine nodes in various network partitions across the bank to achieve concurrent scanning. Although this greatly improves the timeliness of vulnerability detection, the problem of load balancing still exists.
[0004] In the healthcare field, task data scanning is often used for the aggregation and analysis of cross-institutional medical data and privacy compliance checks. For example, in the joint statistics of multi-center clinical data, it is necessary to scan the patient data fragments of each hospital in parallel through distributed nodes. However, existing solutions are limited by fixed fragmentation logic and cannot dynamically adjust the granularity of scanning tasks according to the amount of data in each node. Moreover, if a node fails due to data transmission interruption, a full scan must be restarted instead of resuming the scan from the breakpoint.
[0005] Therefore, existing technologies for task data scanning suffer from problems such as insufficient concurrent scheduling mechanisms and multi-node resource adaptability, lack of dynamic perception and automatic adjustment capabilities in load allocation, failure to achieve fine-grained breakpoint resumption of scanning in fault tolerance and recovery mechanisms, and reliance on manual intervention and lack of flexibility in sharding strategies, which in turn result in low efficiency of task data scanning. Summary of the Invention
[0006] This invention provides a method, apparatus, device, and medium for scanning task data based on byte order fragmentation, in order to solve the technical problem of low efficiency when scanning target task data.
[0007] Firstly, a task data scanning method based on byte order fragmentation is provided, including: Obtain the initial prefix of the target scanning task, and perform byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task; Map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The subtask is assigned to a preset working node, and the working node is used to generate a task scan request corresponding to the subtask; The task area corresponding to the subtask is scanned according to the task scan request, and the task execution progress of the subtask is monitored in real time. Based on the scanning data of the sub-tasks during the task execution progress data collection process; The scan data corresponding to all subtasks are merged into the task scan data of the target scan task.
[0008] Secondly, a task data scanning device based on byte order fragmentation is provided, including: The subprefix set fragmentation expansion module is used to obtain the initial prefix of the target scanning task, perform byte order fragmentation expansion on the initial prefix, and obtain the subprefix set of the target scanning task; The subtask mapping module is used to map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The task scan request generation module is used to assign the subtask to a preset working node and use the working node to generate a task scan request corresponding to the subtask. The task execution progress monitoring module is used to perform data scanning on the task area corresponding to the subtask according to the task scanning request, and to monitor the task execution progress of the subtask in real time. A scan data acquisition module is used to acquire scan data of the sub-tasks during the data scanning process based on the task execution progress; The task scan data merging module is used to merge the scan data corresponding to all subtasks into the task scan data of the target scan task.
[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described byte order fragmentation-based task data scanning method.
[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described byte-order fragmentation-based task data scanning method.
[0011] In the above-described scheme implemented by the byte order fragmentation-based task data scanning method, apparatus, device, and medium, the initial prefix of the target scanning task can be obtained through a client; the initial prefix can be expanded by byte order fragmentation to obtain a set of sub-prefixes for the target scanning task; all sub-prefixes in the set can be mapped to corresponding sub-tasks in the target scanning task; the sub-tasks can be assigned to preset working nodes, and task scanning requests corresponding to the sub-tasks can be generated using the working nodes; data scanning can be performed on the task area corresponding to the sub-tasks according to the task scanning requests, and the task execution progress of the sub-tasks can be monitored in real time; the scanning data of the sub-tasks during the data scanning process can be collected based on the task execution progress; and the scanning data corresponding to all sub-tasks can be merged into the task scanning data of the target scanning task. The task scan data is fed back to the client. In this invention, the initial prefix is split into a set of non-overlapping sub-prefixes through byte order sharding expansion. Combined with subtask mapping and multi-worker node allocation, the limitations of traditional single-threaded scanning are broken, allowing concurrency to scale linearly with the number of worker nodes. This significantly reduces the time consumption of enumerating massive amounts of data. At the same time, the dynamically adapted sharding logic avoids resource waste caused by fixed sharding. Before subtask allocation, the processing capacity is evaluated based on the real-time status of the worker nodes, and a load balancing strategy is used to allocate tasks, effectively avoiding overload of hot nodes and making full use of idle node resources, thereby improving the overall resource utilization of the cluster. Real-time monitoring of subtask execution progress can promptly identify timeout and failed tasks. Timeout tasks are re-sharded, and failed tasks are reassigned to idle nodes to resume scanning. Data collection verification ensures that the scanned data is not duplicated or missing. Sorting and deduplication during the merging stage ensure the integrity and accuracy of the final task data, further improving the efficiency of task data scanning. Attached Figure Description
[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a schematic diagram of an application environment for a task data scanning method based on byte order fragmentation according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a task data scanning method based on byte order fragmentation in one embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 4 yes Figure 2A flowchart illustrating a specific implementation of step S6; Figure 5 This is a schematic diagram of a task data scanning device based on byte order fragmentation in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0015] The task data scanning method based on byte order fragmentation provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the initial prefix of the target scanning task from the client, perform byte order fragmentation and expansion on the initial prefix to obtain a set of sub-prefixes for the target scanning task; map all sub-prefixes in the sub-prefix set to corresponding sub-tasks in the target scanning task; assign the sub-tasks to preset working nodes, and use the working nodes to generate task scanning requests corresponding to the sub-tasks; perform data scanning on the task areas corresponding to the sub-tasks according to the task scanning requests, and monitor the task execution progress of the sub-tasks in real time; collect the scanning data of the sub-tasks during the data scanning process based on the task execution progress; and merge the scanning data corresponding to all sub-tasks into the task scanning data of the target scanning task. The task scan data is fed back to the client. In this invention, the initial prefix is split into a set of non-overlapping sub-prefixes through byte order sharding expansion. Combined with subtask mapping and multi-worker node allocation, the limitations of traditional single-threaded scanning are broken, allowing concurrency to scale linearly with the number of work nodes, significantly reducing the time consumption for enumerating massive amounts of data. Simultaneously, the dynamically adaptable sharding logic avoids resource waste caused by fixed sharding. Before subtask allocation, the processing capacity is assessed based on the real-time status of the work nodes, and a load balancing strategy is used to allocate tasks, effectively avoiding overload of hot nodes and fully utilizing the resources of idle nodes, thus improving the overall resource utilization of the cluster. Real-time monitoring of subtask execution progress can promptly identify timeout and failed tasks. Timeout tasks are re-sharded, and failed tasks are reassigned to idle nodes to resume scanning. Data collection verification ensures that scanned data is not duplicated or missing, and sorting and deduplication during the merging stage ensure the integrity and accuracy of the final task data, further improving the efficiency of task data scanning. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.
[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a task data scanning method based on byte order fragmentation provided in an embodiment of the present invention includes the following steps: S1. Obtain the initial prefix of the target scanning task, and perform byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task.
[0017] In this embodiment of the invention, the target scanning task refers to a complete data enumeration operation initiated by a user or system in a distributed key-value storage system, which aims to traverse and obtain all key-value pairs that meet a specific prefix condition. It is typically used in scenarios such as data migration, backup, and analysis.
[0018] In detail, the initial prefix refers to the starting string specified in the target scanning task for filtering key-value pairs. The system will perform a limited scan of the key space based on this prefix. For example, "user_" means all keys that start with "user_".
[0019] In this embodiment of the invention, the step of performing byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task includes: The last byte of the initial prefix is segmented and expanded to obtain multiple sub-prefixes; Determine whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task; If the multiple sub-prefixes need to be expanded at multiple levels, then identify the target sub-prefix among the multiple sub-prefixes that needs to be expanded at multiple levels; The last byte in the target sub-prefix is segmented and expanded to obtain multiple second-level sub-prefixes of the target sub-prefix, and then the process returns to the step of determining whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task, until the multiple sub-prefixes no longer need to be expanded at multiple levels. If the multiple sub-prefixes do not require multi-level expansion, then the sub-prefix set of the target scanning task is determined based on the multiple sub-prefixes and the multiple secondary sub-prefixes.
[0020] In detail, the initial prefix's byte structure is first parsed to locate the binary storage value of the last byte. Then, according to the byte's value range (0-255, corresponding to hexadecimal 00-FF), 256 different byte values are generated. Next, each generated byte value is sequentially concatenated after the last byte of the initial prefix, replacing the original last byte, forming 256 new character sequences (i.e., sub-prefixes). For example, when the initial prefix is "[A,B,C]" (byte sequence), it is expanded to "[A,B,C,0]、[A,B,C,1]、...、[A,B,C,255]". Finally, logical checks ensure that the key spaces of all sub-prefixes do not overlap and completely cover the key range corresponding to the initial prefix, avoiding data omissions or duplicate overwriting.
[0021] Specifically, first, the preset concurrency requirement of the target scanning task is obtained (i.e., the number of subtasks expected to be executed simultaneously, usually matching the number of available worker nodes in the cluster); then, the total number of currently generated sub-prefixes is counted (e.g., 256 after first-level expansion); then, the total number of sub-prefixes is compared with the preset concurrency. If the total number of sub-prefixes is less than the preset concurrency (e.g., the preset concurrency is 500, and there are only 256 sub-prefixes), it is determined that multi-level expansion is required; if the total number of sub-prefixes is greater than or equal to the preset concurrency, it is determined that multi-level expansion is not required.
[0022] Further, from the current subprefix set, subprefixes that need multi-level expansion are selected (usually all subprefixes when the total number of subprefixes does not meet the concurrency requirements, or some subprefixes with excessive data volume). Then, for each selected subprefix, its byte structure is parsed and the position of the next byte after the current last byte is located (e.g., the next byte position of the subprefix "[A,B,C,0]" is after the 4th byte). Then, 256 byte values are generated according to the byte value range (0-255), and concatenated to the end of the subprefix in sequence to form 256 secondary subprefixes (e.g., "[A,B,C,0,0]、[A,B,C,0,1]、...、[A,B,C,0,255]"). After that, all secondary subprefixes are integrated into a new subprefix set, covering the original subprefixes that need expansion. Finally, the computer re-executes the step of "determining whether multiple subprefixes need multi-level expansion based on the concurrency of the target scanning task", repeating the above process until the total number of subprefixes meets the concurrency requirements or the data volume is balanced.
[0023] Furthermore, a unique identifier (such as a subprefix ID) is assigned to each of the currently generated subprefixes (including subprefixes obtained after multi-level expansion), and the start key and end key corresponding to each subprefix are recorded. All subprefixes with unique identifiers and key range information are organized into a structured list to form the subprefix set of the target scanning task.
[0024] The subprefix set refers to a structured data set containing multiple subprefixes obtained by byte order fragmentation expansion (one or more levels) of the initial prefix. Each subprefix corresponds to an independent and non-overlapping subrange in the target scan task key space, and the sum of the key ranges of all subprefixes in the set completely covers the target key space corresponding to the initial prefix.
[0025] In this embodiment of the invention, the initial prefix of the target scanning task is obtained and its byte order is split and expanded to obtain a set of sub-prefixes. The advantage is that it can break the limitation of traditional fixed prefix single-threaded scanning. By leveraging the natural order of byte order, it achieves non-overlapping and accurate splitting of sub-prefixes, ensuring data coverage integrity while supporting on-demand expansion of concurrency, improving the efficiency of scanning massive amounts of data, and without requiring modification of the core logic of the storage engine, thus exhibiting strong compatibility.
[0026] In the fintech field, when backing up customer transaction data in banking systems, the initial prefix "trans_2025" (transaction data in 2025) is used to generate a set of sub-prefixes. This allows the backup task to be split into 256 or more sub-tasks to be executed concurrently. The full backup that originally took 24 hours can be shortened to within 2 hours, avoiding excessive resource consumption by the backup task during peak trading periods.
[0027] In the healthcare field, when summarizing and analyzing patient medical record data across hospitals, a set of subprefixes is generated using "patient_hospitalA" (patient data from hospital A) as the initial prefix. This allows medical record scanning tasks to be distributed to multiple nodes for parallel processing, quickly completing compliance checks and statistical analysis of medical record data. Furthermore, the non-overlapping nature of the subprefixes ensures that patient data is not duplicated or omitted, guaranteeing the accuracy of medical data.
[0028] S2. Map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task.
[0029] In this embodiment of the invention, the sub-prefix is a string obtained by byte order fragmentation expansion of the initial prefix, used to limit the scanning range of each subtask.
[0030] In detail, a subtask refers to the smallest independently executable scanning unit into which the target scanning task is split according to the subprefixes in the subprefix set. Each subtask corresponds one-to-one with a subprefix and includes information such as subtask ID, corresponding subprefix, scanning range (start and end keys of the subprefix), and execution priority.
[0031] Specifically, the subprefix set is read from storage, each subprefix in the set is traversed, and the unique identifier, start key, and end key of each subprefix are extracted. Then, the subtask generation module is called to create a subtask object for each subprefix. The unique identifier of the subprefix is used as the core part of the subtask ID, and the start key and end key of the subprefix are used as the scan range parameters of the subtask. At the same time, the execution priority of the subtask is set according to business requirements. All subtasks are organized into a subtask list and stored in the task scheduling queue, thus completing the mapping from subprefix to subtask.
[0032] In this embodiment of the invention, all subprefixes in the subprefix set are mapped to corresponding subtasks. The advantage is that the abstract subprefixes can be transformed into executable specific task units, providing a basis for subsequent task allocation and concurrent execution. The one-to-one mapping relationship ensures that the scanning task of each subprefix is not repeated or omitted. At the same time, the independent attributes of the subtasks support flexible priority scheduling and fault recovery.
[0033] In the fintech field, when applying real-time market data analysis tasks to securities trading systems, after mapping the subprefix set generated by the initial prefix "market_quote" to subtasks, the priority of subtasks can be set according to the importance of market data (e.g., the priority of the subtask for large-cap stocks is higher than that for small-cap stocks), ensuring that key market data is analyzed first and providing timely support for trading decisions.
[0034] In the healthcare field, when monitoring the operational data of medical equipment in hospitals, the "device_ECG" (electrocardiogram device data) sub-prefix set is mapped to sub-tasks. Each sub-task corresponds to the data scan of one or more ECG devices. If the data of a certain device is abnormal, only the corresponding sub-task needs to be re-executed, without restarting the entire monitoring task, thus improving the efficiency of fault handling.
[0035] S3. Assign the subtask to a preset working node, and use the working node to generate a task scan request corresponding to the subtask.
[0036] In this embodiment of the invention, the preset working node refers to the computing node pre-configured in the distributed cluster for performing subtask scanning operations. These nodes have the ability to communicate with distributed key-value storage engines (such as TiKV and HBase), possess certain CPU, memory, and network bandwidth resources, and are included in the management scope of the task scheduling system. They can receive and execute subtasks, generate task scanning requests, and interact with storage nodes to obtain data.
[0037] In detail, a task scan request is a structured request generated by a worker node based on subtask information. It is used to request the distributed key-value storage engine to read data of the corresponding key range. It includes parameters such as request ID, subtask ID, scan start key, scan end key, data read batch size, and task execution progress (such as CheckPoint progress).
[0038] In the embodiments of the present invention, see Figure 3 As shown, assigning the subtask to a preset working node includes: S31. Obtain the real-time status information of the preset working nodes; S32. Analyze the task processing capability of the working node based on the real-time status information; S33. Assign the subtasks to the working nodes according to the task processing capacity.
[0039] In detail, the status data of each working node, such as CPU utilization, memory usage, network bandwidth, and current task count, are obtained periodically through a heartbeat mechanism or resource monitoring interface.
[0040] Specifically, the status parameters of each valid worker node are read from the real-time status database, and a weight is assigned to each parameter (e.g., CPU utilization weight 0.3, memory usage weight 0.3, network bandwidth utilization weight 0.2, number of executed subtasks weight 0.1, and task execution success rate weight 0.1). Each parameter is then standardized, converting the parameter value into a score of 0-100 (e.g., lower CPU utilization, lower memory usage, and higher task execution success rate all result in higher scores). Then, a comprehensive score is calculated for each worker node based on the weights. The task processing capability is categorized according to the comprehensive score: 80-100 indicates high processing capability (multiple subtasks can be assigned), 50-79 indicates medium processing capability (1-2 subtasks can be assigned), and 0-49 indicates low processing capability (no subtasks assigned). The maximum number of subtasks each node can handle is also recorded (e.g., a high-capacity node can handle a maximum of 10 subtasks, and a medium-capacity node can handle a maximum of 3 subtasks).
[0041] Furthermore, extract the subtasks to be assigned from the subtask queue, sort them according to their priority (higher priority is assigned first), and then traverse all valid working nodes, prioritizing nodes with high task processing capacity and whose current number of subtasks has not reached the maximum capacity.
[0042] In this embodiment of the invention, subtasks are assigned to preset working nodes and task scanning requests are generated using the working nodes. The advantage is that tasks can be dynamically allocated based on the real-time load of the working nodes, avoiding overload of hot nodes and improving the utilization of cluster resources. In addition, the working nodes generate scanning requests that conform to the storage engine protocol, ensuring that data scanning can be executed smoothly and reducing retries caused by incorrect request formats.
[0043] In the fintech field, when used for transaction reconciliation tasks on payment platforms, the processing capacity of worker nodes is analyzed in real time, and reconciliation sub-tasks are distributed to nodes with lower loads. This avoids delays in payment transactions caused by a single node handling a large number of reconciliation tasks. At the same time, worker nodes generate scan requests that conform to the TiKV protocol to quickly read transaction data and improve reconciliation efficiency.
[0044] In the healthcare field, when performing compliance review tasks for medical insurance reimbursement data, the medical insurance data scanning sub-tasks from different regions are assigned to the corresponding work nodes to reduce cross-regional network transmission latency.
[0045] S4. Perform data scanning on the task area corresponding to the subtask according to the task scanning request, and monitor the task execution progress of the subtask in real time.
[0046] In this embodiment of the invention, the task region refers to the data storage area in the distributed key-value storage system that matches the key range covered by the sub-prefix corresponding to the sub-task. It is usually managed by a specific storage node (such as the Region node of TiKV or the RegionServer of HBase). This region stores all the key-value data that the sub-task needs to scan. Its boundary is defined by the start key and end key of the sub-task, ensuring that the sub-task scan is limited to the target data region.
[0047] In this embodiment of the invention, the step of scanning the task region corresponding to the subtask according to the task scanning request includes: The parameter format of the task scan request is validated. If the format is incorrect, the update task scan request corresponding to the subtask is regenerated. If the format is correct, obtain the task region corresponding to the subtask in the task scan request; The task region is scanned according to the byte order in the subprefix set, and the target key-value data in the task region is read.
[0048] In detail, check whether the start key and end key in the task scan request are valid (e.g., not empty, correct length, consistent encoding). If they do not meet the requirements, reconstruct the request parameters.
[0049] Specifically, the start key and end key in the request are extracted, and the storage node (Region node or RegionServer) to which the start key and end key belong is queried through the routing table of the distributed key-value storage system (such as the Region routing table of TiKV or the Meta table of HBase). The IP address and port number of the storage node are obtained, and the key range managed by the storage node is the task region corresponding to the subtask.
[0050] Furthermore, the worker node initiates a Scan operation to the storage engine, traversing all key-value pairs within the task area in ascending byte order and returning the results one by one. Following the ascending byte order in the sub-prefix set (from the start key to the end key), key-value data is retrieved from the local storage engine (such as RocksDB). During retrieval, the system first searches the MemTable in memory, then packages the retrieved key-value data into packets, each packet containing a batch number and a list of key-value pairs.
[0051] The target key-value data refers to the key-value data within the current subtask scanning range (defined by the start key and end key corresponding to the sub-prefix). This data includes keys (such as user ID, transaction ID, etc.) and corresponding values (such as user information, transaction details, etc.) in specific business scenarios.
[0052] In this embodiment of the invention, it refers to the proportion of the amount of key-value data that has been scanned in the subtask during the data scanning process to the total amount of data that the subtask needs to scan, or the proportion of the range of keys that have been scanned to the total range of keys in the subtask. It is usually quantified by indicators such as "the last key that has been scanned", "the number of data entries that have been scanned", and "time taken". Its core function is to provide a basis for real-time monitoring of the subtask status, judging whether it has timed out or failed, and triggering fault recovery.
[0053] In this embodiment of the invention, the real-time monitoring of the task execution progress of the subtask includes: Obtain the actual execution time of all subtasks and calculate the average execution time of all subtasks; Compare the actual execution time of a single subtask with the average execution time; If the actual execution time of a single subtask exceeds a preset multiple of the average execution time, the execution progress of the single subtask will be marked as a timeout. If the actual execution time of a single subtask exceeds a preset multiple of the average execution time and the working node does not report the task execution progress of the subtask, the task execution progress of the single subtask will be marked as a task failure. If the actual execution time of a single subtask does not exceed a preset multiple of the average execution time, the execution progress of the single subtask is marked as successful.
[0054] In detail, the coordinating node obtains the status information of all executing subtasks from each worker node through periodic polling (e.g., every 30 seconds). The status information includes the subtask ID, start time, the last key scanned, and the number of data entries scanned. For each subtask, the coordinating node calculates its actual execution time by subtracting the start time from the current system time. Next, it filters out all subtasks in the "running" state, sums their actual execution times, and divides the total execution time by the number of "running" subtasks to obtain the average execution time for all subtasks. If the number of "running" subtasks is 0, the average execution time is set to a preset default value (e.g., 5 minutes).
[0055] Specifically, the coordinating node reads the latest average execution time and obtains the latest actual execution time of each "in-process" subtask from the worker nodes. For each subtask, it calculates the ratio of its actual execution time to the average execution time (e.g., if a subtask's actual execution time is 10 minutes and its average execution time is 5 minutes, the ratio is 2). Next, the coordinating node reads a preset comparison threshold (e.g., a ratio threshold of 2.5, meaning that actual execution time exceeding 2.5 times the average execution time requires special attention). Then, it compares the execution time ratio of each subtask with the threshold, recording subtasks with ratios less than the threshold as "task successful" and subtasks with ratios greater than or equal to the threshold as "suspected timeout". Simultaneously, the coordinating node also considers the percentage of scanned keys in the subtask's key range (scanned key range / total key range). If a subtask's execution time ratio exceeds the threshold but the scanned key range is above 90%, it is not marked as abnormal. If the scanned key range is below 50% and the execution time ratio exceeds the threshold, it is marked as "task timeout".
[0056] Furthermore, after marking a subtask as "suspected timeout", a progress report request is sent to the corresponding worker node, requiring the worker node to provide feedback on the latest task execution progress within a preset time (e.g., 10 seconds). If no response (including progress report and timeout reason feedback) is received from the worker node after the preset time, the worker node is determined to be faulty, and the subtask cannot continue to be executed due to the node failure. Its task execution progress is marked as "task failed".
[0057] In this embodiment of the invention, data scanning of the task area is performed according to the task scanning request and the task execution progress is monitored in real time. The advantage is that it can ensure that the data scanning is carried out in byte order, avoid data reading chaos, and at the same time, timeout and failed tasks can be detected in time through real-time progress monitoring, providing a basis for subsequent fault tolerance processing and reducing resource waste.
[0058] In the fintech field, when scanning data for net asset value (NAV) calculation in fund systems, scanning in byte order ensures the correct temporal sequence of fund transaction data. Real-time progress monitoring can promptly detect when a particular fund's data scan times out, avoiding impacting the overall NAV calculation timeliness and ensuring timely release of fund NAVs.
[0059] In the healthcare field, when performing traceability scanning tasks for vaccination records, orderly scanning ensures that vaccination records are read in chronological order, facilitating the tracing of the vaccination process. Real-time progress monitoring can quickly identify scanning failures in a particular region, allowing for timely task reassignment and ensuring that vaccine traceability tasks are completed on time, thus safeguarding public health security.
[0060] S5. Scanning data of the sub-tasks during the data scanning process based on the task execution progress.
[0061] In this embodiment of the invention, scan data refers to the set of key-value data that conforms to the scan range of the subtask, which is read from the distributed key-value storage system when the subtask performs data scanning within the task area.
[0062] In this embodiment of the invention, the scanning data of the subtask during the task execution progress data collection scanning process includes: If the task execution progress is a task timeout, the subtask corresponding to the task timeout is taken as the target scanning task, and the process returns to the step of byte order fragmentation expansion of the initial prefix until the task execution progress is a task success. If the task execution progress is "task failed", the subtask corresponding to the task failure is assigned to an idle worker node, and the process returns to the step of generating the task scan request corresponding to the subtask using the worker node, until the task execution progress is "task successful". If the task execution progress indicates a successful task, the target key-value data read during the data scanning process will be used as the scanning data for the subtask.
[0063] In detail, if the task execution progress is a task timeout, the current scan position of the timed-out subtask is used as a new initial prefix, and byte order fragmentation expansion is performed again to generate finer-grained subtasks to improve concurrency efficiency.
[0064] Specifically, extract the failed subtasks, read the task execution progress of the subtask (e.g., "[A,B,C,1,10]", meaning the key was scanned before the failure), query the currently "idle" worker nodes in the cluster (with a comprehensive score of 80-100 and no unfinished subtasks), select an idle worker node, reallocate the parameters of the failed subtask (the scan range is set to the key of the task execution progress + 1 to the original end key, the subtask ID remains unchanged, and the priority remains unchanged) to the node, and regenerate the task scan request to continue execution.
[0065] Furthermore, if the task execution progress indicates a successful task, all key-value pairs obtained from the storage engine during the subtask execution process are saved as scan data results.
[0066] In this embodiment of the invention, differentiated processing is adopted for different task execution progress (timeout, failure, success) to ensure that timeout tasks can be accelerated by re-sharding, failed tasks can be quickly recovered by reallocation, and the scanning data of successful tasks can be completely collected, thus ensuring the continuity and data integrity of the overall scanning task.
[0067] In the fintech field, when scanning position data in a securities clearing system, if the scanning of a brokerage's position data times out, the remaining data is scanned by resharding to avoid delays in the clearing task. If a node's scanning task fails, it is reassigned to an idle node to continue scanning, ensuring that no clearing data is missing. Successfully collected position data provides an accurate basis for subsequent clearing calculations.
[0068] In the healthcare field, when scanning data for archiving physical examination reports, timed-out scans are quickly completed by reslicing the remaining reports. Failed tasks are reassigned to avoid missing reports, and successfully collected reports are archived according to standards, providing complete data support for subsequent patient health inquiries and statistical analysis.
[0069] S6. Merge the scan data corresponding to all subtasks into the task scan data of the target scan task.
[0070] In this embodiment of the invention, task scan data refers to the set of all key-value pairs finally obtained by the target scan task, which is formed by sequentially merging the scan data of each subtask.
[0071] In the embodiments of the present invention, see Figure 4 As shown, merging the scan data corresponding to all subtasks into the task scan data of the target scan task includes: S41. Sort the scan data corresponding to all subtasks according to the byte order of the subprefixes in the subprefix set to obtain sorted scan data; S42. Perform deduplication processing on the sorted scan data to obtain deduplicated scan data; S43. Consolidate the deduplicated scan data of all subtasks into the task scan data of the target scan task.
[0072] In detail, the byte order of each subprefix in the subprefix set is read (e.g., the order of subprefixes "[A,B,C,0]", "[A,B,C,1]", and "[A,B,C,2]"); a corresponding subprefix sorting identifier is added to the scan data of each subtask (e.g., the sorting identifier for subprefix "[A,B,C,0]" is 1, and for "[A,B,C,1]" it is 2); the scan data of all subtasks are initially sorted according to the ascending order of the sorting identifiers to ensure that the data order at the subtask level is consistent with the subprefix byte order, and then arranged into a continuous data stream according to the subprefix byte order to form sorted scan data.
[0073] Specifically, iterate through the sorted list of key-value pairs, compare adjacent keys for duplicates, and if duplicates are found, keep the first key-value pair that appears and delete subsequent duplicates.
[0074] Furthermore, the coordinating node sequentially concatenates the deduplicated scan data into a continuous binary data stream or structured file. The file contains metadata about the data (target scan task ID, scan start time, scan end time, total number of data entries, and number of deduplicated entries). The concatenated data stream or file is marked as the task scan data of the target scan task, stored in the final result storage area, and a task completion notification is sent to the client, informing them of the result storage address.
[0075] In this embodiment of the invention, the correctness of the data order is ensured by sorting by byte order, duplicate data is eliminated by deduplication, and complete result data is formed by splicing, providing a high-quality data foundation for subsequent operation and maintenance. Moreover, the integrity verification during the merging process can detect data vulnerabilities in a timely manner and ensure data reliability.
[0076] In the fintech field, when used for bank customer asset data migration tasks, the merged task scan data is sorted by customer ID byte order, with no duplicate asset records, ensuring that the customer asset data migrated to the new system is accurate and avoiding asset statistical errors.
[0077] In the healthcare field, when used for electronic medical record aggregation tasks on regional healthcare platforms, the merged task scan data is sorted by patient ID byte order, duplicate medical record records are removed, providing doctors with complete and accurate patient history medical records to assist in accurate diagnosis, while complying with medical data compliance requirements and avoiding the waste of storage resources caused by data redundancy.
[0078] As can be seen, the above scheme utilizes byte order sharding to generate a set of non-overlapping sub-prefixes, combined with subtask mapping and load balancing, to overcome the limitations of single-threaded scanning. The concurrency scales linearly with the number of worker nodes, significantly improving the efficiency of scanning massive amounts of data. Real-time progress monitoring and the continuation scanning mechanism prevent full retries in case of task failure, ensuring reliability. The sorting and deduplication during the merging phase ensures data integrity and accuracy, and can efficiently support data migration, backup, and other operational scenarios, thereby improving the efficiency of task data scanning.
[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0080] In one embodiment, a task data scanning device based on byte order fragmentation is provided, which corresponds one-to-one with the task data scanning method based on byte order fragmentation in the above embodiments. For example... Figure 5As shown, the byte order-based task data scanning device 100 includes a sub-prefix set fragmentation extension module 101, a sub-task mapping module 102, a task scan request generation module 103, a task execution progress monitoring module 104, a scan data acquisition module 105, and a task scan data merging module 106. Detailed descriptions of each functional module are as follows: The subprefix set fragmentation expansion module 101 is used to obtain the initial prefix of the target scanning task, perform byte order fragmentation expansion on the initial prefix, and obtain the subprefix set of the target scanning task; Subtask mapping module 102 is used to map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The task scan request generation module 103 is used to assign the subtask to a preset working node and use the working node to generate a task scan request corresponding to the subtask. The task execution progress monitoring module 104 is used to perform data scanning on the task area corresponding to the sub-task according to the task scanning request, and to monitor the task execution progress of the sub-task in real time. The scanning data acquisition module 105 is used to acquire the scanning data of the sub-task during the data scanning process based on the task execution progress; The task scan data merging module 106 is used to merge the scan data corresponding to all subtasks into the task scan data of the target scan task.
[0081] In one embodiment, the subprefix set fragmentation expansion module 101, when performing byte order fragmentation expansion on the initial prefix to obtain the subprefix set of the target scanning task, is used to: The last byte of the initial prefix is segmented and expanded to obtain multiple sub-prefixes; Determine whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task; If the multiple sub-prefixes need to be expanded at multiple levels, then identify the target sub-prefix among the multiple sub-prefixes that needs to be expanded at multiple levels; The last byte in the target sub-prefix is segmented and expanded to obtain multiple second-level sub-prefixes of the target sub-prefix, and then the process returns to the step of determining whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task, until the multiple sub-prefixes no longer need to be expanded at multiple levels. If the multiple sub-prefixes do not require multi-level expansion, then the sub-prefix set of the target scanning task is determined based on the multiple sub-prefixes and the multiple secondary sub-prefixes.
[0082] In one embodiment, the task scan request generation module 103, when performing the assignment of the subtask to a preset working node, is used to: Obtain real-time status information of preset working nodes; The task processing capability of the working node is analyzed based on the real-time status information. The subtasks are assigned to the work nodes based on the task processing capacity.
[0083] In one embodiment, the task execution progress monitoring module 104, when performing data scanning on the task area corresponding to the subtask according to the task scanning request, is used to: The parameter format of the task scan request is validated. If the format is incorrect, the update task scan request corresponding to the subtask is regenerated. If the format is correct, obtain the task region corresponding to the subtask in the task scan request; The task region is scanned according to the byte order in the subprefix set, and the target key-value data in the task region is read.
[0084] In one embodiment, the task execution progress monitoring module 104, when performing real-time monitoring of the task execution progress of the sub-task, is used to: Obtain the actual execution time of all subtasks and calculate the average execution time of all subtasks; Compare the actual execution time of a single subtask with the average execution time; If the actual execution time of a single subtask exceeds a preset multiple of the average execution time, the execution progress of the single subtask will be marked as a timeout. If the actual execution time of a single subtask exceeds a preset multiple of the average execution time and the working node does not report the task execution progress of the subtask, the task execution progress of the single subtask will be marked as a task failure. If the actual execution time of a single subtask does not exceed a preset multiple of the average execution time, the execution progress of the single subtask is marked as successful.
[0085] In one embodiment, the scan data acquisition module 105, when performing scan data acquisition of the subtask during the task execution progress scan process, is used to: If the task execution progress is a task timeout, the subtask corresponding to the task timeout is taken as the target scanning task, and the process returns to the step of byte order fragmentation expansion of the initial prefix until the task execution progress is a task success. If the task execution progress is "task failed", the subtask corresponding to the task failure is assigned to an idle worker node, and the process returns to the step of generating the task scan request corresponding to the subtask using the worker node, until the task execution progress is "task successful". If the task execution progress indicates a successful task, the target key-value data read during the data scanning process will be used as the scanning data for the subtask.
[0086] In one embodiment, the task scan data merging module 106, when performing the task scan data merging of all subtasks into the target scan task, is used to: The scan data corresponding to all subtasks are sorted according to the byte order of the subprefixes in the subprefix set to obtain sorted scan data; The sorted scan data is deduplicated to obtain deduplicated scan data; The deduplicated scan data of all subtasks are concatenated to form the task scan data of the target scan task.
[0087] This invention provides a task data scanning device based on byte order sharding. It utilizes byte order sharding to expand and generate a set of non-overlapping sub-prefixes, combined with sub-task mapping and load balancing, to overcome the limitations of single-threaded scanning. The concurrency scales linearly with the number of working nodes, significantly improving the efficiency of scanning massive amounts of data. Real-time progress monitoring and a resume scanning mechanism prevent full retries in case of task failure, ensuring reliability. The sorting and deduplication during the merging phase ensures data integrity and accuracy, and can efficiently support data migration, backup, and other operation and maintenance scenarios, thereby improving the efficiency of task data scanning.
[0088] Specific limitations regarding the byte-order fragmentation-based task data scanning device can be found in the limitations of the byte-order fragmentation-based task data scanning method described above, and will not be repeated here. Each module in the aforementioned byte-order fragmentation-based task data scanning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.
[0089] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a byte-order fragmentation-based task data scanning method on the server side.
[0090] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a byte-order fragmentation-based task data scanning method.
[0091] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial prefix of the target scanning task, and perform byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task; Map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The subtask is assigned to a preset working node, and the working node is used to generate a task scan request corresponding to the subtask; The task area corresponding to the subtask is scanned according to the task scan request, and the task execution progress of the subtask is monitored in real time. Based on the scanning data of the sub-tasks during the task execution progress data collection process; The scan data corresponding to all subtasks are merged into the task scan data of the target scan task.
[0092] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the initial prefix of the target scanning task, and perform byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task; Map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The subtask is assigned to a preset working node, and the working node is used to generate a task scan request corresponding to the subtask; The task area corresponding to the subtask is scanned according to the task scan request, and the task execution progress of the subtask is monitored in real time. Based on the scanning data of the sub-tasks during the task execution progress data collection process; The scan data corresponding to all subtasks are merged into the task scan data of the target scan task.
[0093] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0095] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0096] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.
[0097] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A task data scanning method based on byte order fragmentation, characterized in that, include: Obtain the initial prefix of the target scanning task, and perform byte order fragmentation expansion on the initial prefix to obtain the sub-prefix set of the target scanning task; Map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The subtask is assigned to a preset working node, and the working node is used to generate a task scan request corresponding to the subtask; The task area corresponding to the subtask is scanned according to the task scan request, and the task execution progress of the subtask is monitored in real time. Based on the scanning data of the sub-tasks during the task execution progress data collection process; The scan data corresponding to all subtasks are merged into the task scan data of the target scan task.
2. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The byte order fragmentation expansion of the initial prefix to obtain the sub-prefix set of the target scanning task includes: The last byte of the initial prefix is segmented and expanded to obtain multiple sub-prefixes; Determine whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task; If the multiple sub-prefixes need to be expanded at multiple levels, then identify the target sub-prefix among the multiple sub-prefixes that needs to be expanded at multiple levels; The last byte in the target sub-prefix is segmented and expanded to obtain multiple second-level sub-prefixes of the target sub-prefix, and then the process returns to the step of determining whether the multiple sub-prefixes need to be expanded at multiple levels based on the concurrency of the target scanning task, until the multiple sub-prefixes no longer need to be expanded at multiple levels. If the multiple sub-prefixes do not require multi-level expansion, then the sub-prefix set of the target scanning task is determined based on the multiple sub-prefixes and the multiple secondary sub-prefixes.
3. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The step of assigning the subtask to a preset working node includes: Obtain real-time status information of preset working nodes; The task processing capability of the working node is analyzed based on the real-time status information. The subtasks are assigned to the work nodes based on the task processing capacity.
4. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The step of scanning the task area corresponding to the subtask according to the task scanning request includes: The parameter format of the task scan request is validated. If the format is incorrect, the update task scan request corresponding to the subtask is regenerated. If the format is correct, obtain the task region corresponding to the subtask in the task scan request; The task region is scanned according to the byte order in the subprefix set, and the target key-value data in the task region is read.
5. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The real-time monitoring of the task execution progress of the subtask includes: Obtain the actual execution time of all subtasks and calculate the average execution time of all subtasks; Compare the actual execution time of a single subtask with the average execution time; If the actual execution time of a single subtask exceeds a preset multiple of the average execution time, the execution progress of the single subtask will be marked as a timeout. If the actual execution time of a single subtask exceeds a preset multiple of the average execution time and the working node does not report the task execution progress of the subtask, the task execution progress of the single subtask will be marked as a task failure. If the actual execution time of a single subtask does not exceed a preset multiple of the average execution time, the execution progress of the single subtask is marked as successful.
6. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The scanning data of the subtasks during the scanning process based on the task execution progress data collection includes: If the task execution progress is a task timeout, the subtask corresponding to the task timeout is taken as the target scanning task, and the process returns to the step of byte order fragmentation expansion of the initial prefix until the task execution progress is a task success. If the task execution progress is "task failed", the subtask corresponding to the task failure is assigned to an idle worker node, and the process returns to the step of generating the task scan request corresponding to the subtask using the worker node, until the task execution progress is "task successful". If the task execution progress indicates a successful task, the target key-value data read during the data scanning process will be used as the scanning data for the subtask.
7. The task data scanning method based on byte order fragmentation as described in claim 1, characterized in that, The step of merging the scan data corresponding to all subtasks into the task scan data of the target scan task includes: The scan data corresponding to all subtasks are sorted according to the byte order of the subprefixes in the subprefix set to obtain sorted scan data; The sorted scan data is deduplicated to obtain deduplicated scan data; The deduplicated scan data of all subtasks are concatenated to form the task scan data of the target scan task.
8. A task data scanning device based on byte order fragmentation, characterized in that, include: The subprefix set fragmentation expansion module is used to obtain the initial prefix of the target scanning task, perform byte order fragmentation expansion on the initial prefix, and obtain the subprefix set of the target scanning task; The subtask mapping module is used to map all subprefixes in the subprefix set to the corresponding subtasks in the target scanning task; The task scan request generation module is used to assign the subtask to a preset working node and use the working node to generate a task scan request corresponding to the subtask. The task execution progress monitoring module is used to perform data scanning on the task area corresponding to the subtask according to the task scanning request, and to monitor the task execution progress of the subtask in real time. A scan data acquisition module is used to acquire scan data of the sub-tasks during the data scanning process based on the task execution progress; The task scan data merging module is used to merge the scan data corresponding to all subtasks into the task scan data of the target scan task.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the task data scanning method based on byte order fragmentation as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the task data scanning method based on byte order fragmentation as described in any one of claims 1 to 7.