Dynamic threshold-based file chunking parallel processing method and parallel processing system

Abstract

The application relates to the technical field of data processing, in particular to a file block parallel processing method and a parallel processing system based on a dynamic threshold value. The method comprises the following steps: identifying a processing parameter of a to-be-processed file at a current time and a data feature of the to-be-processed file, and determining a block processing threshold value of the to-be-processed file; when a to-be-processed data amount of the to-be-processed file is greater than the block processing threshold value, performing block processing on the to-be-processed file based on the data feature and a preset physical storage page; generating a block execution instruction according to a block feature and an adaptive parameter of each target block, and obtaining an execution block corresponding to each target file block; when it is detected that all the target file blocks complete respective corresponding parallel processing, performing back-falling processing on each execution block according to a block position of each target file block in the to-be-processed file, and obtaining an execution file corresponding to a to-be-transmitted file. The application facilitates improving the efficiency and accuracy in a file packet processing process with a large data amount.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a file block parallel processing method and parallel processing system based on dynamic thresholds. Background Technology

[0002] In today's era where the digital economy is deeply integrated into all sectors, large-volume files have become the core data carriers for scenarios such as enterprise R&D, media creation, scientific research and innovation, and government affairs processing. The processing efficiency and reliability of these large files directly determine the smoothness of business processes and the cost of resource utilization.

[0003] Conventional methods for handling large files primarily involve full-scale serial processing. This typically involves loading a large file entirely into memory or local storage and then executing processing tasks sequentially using a single thread. However, this approach may exceed hardware resource limits due to the full loading of large files. Even if the hardware resource limits are not exceeded, the hardware resources may remain at full capacity for extended periods, creating a single bottleneck where processing a large file consumes all hardware resources. This can lead to uncontrolled resource consumption and low efficiency, preventing other business operations from running in parallel. Furthermore, resource contention can cause overall system response delays or even business interruptions. Summary of the Invention

[0004] To improve the efficiency and accuracy of processing large amounts of data in file packages, this application provides a file block parallel processing method and parallel processing system based on dynamic thresholds.

[0005] Firstly, this application provides a file block parallel processing method based on dynamic thresholds, employing the following technical solution: A file chunking parallel processing method based on dynamic thresholds includes: Identify the processing parameters and data characteristics of the file to be processed at the current time, and determine the block processing threshold of the file to be processed based on the processing parameters and data characteristics; When the amount of data to be processed is greater than the block processing threshold, the file to be processed is divided into blocks based on the data characteristics and the preset physical storage pages to obtain multiple target file blocks. Identify the block characteristics of each target file block, generate block execution instructions corresponding to each target file block based on the block characteristics and adaptive parameters, and obtain the execution block corresponding to each target file block. The adaptive parameters are determined based on the processing parameters at the current time, and the execution block is the execution result obtained after the target file block is processed in parallel according to the corresponding block execution instructions. Once all target file blocks have completed their respective parallel processing, the execution blocks are rolled back according to their positions in the file to be processed, thus obtaining the execution file corresponding to the file to be transmitted.

[0006] By adopting the above technical solution, the processing parameters and data characteristics of the file to be processed are analyzed to determine the block judgment conditions of the file to be processed, which facilitates the improvement of the effectiveness of the block processing results. Block processing based on data characteristics and preset physical storage pages facilitates the improvement of the accuracy of the block processing results. By analyzing the block characteristics of each target block and the corresponding adaptive parameters, it is convenient to generate differentiated block execution instructions for different target file blocks. While meeting diverse processing needs, it is also convenient to improve processing efficiency. By recording the block position of the target file blocks in the file to be processed, it is convenient to ensure that all target file blocks can be accurately reassembled in their original order after completing their respective execution tasks. This avoids file corruption caused by disordered order in traditional block processing, thereby improving the accuracy in the processing of large data packages.

[0007] In one possible implementation, the step of dividing the file to be processed into blocks based on the data characteristics and preset physical storage pages to obtain multiple target file blocks includes: Based on the data characteristics, the indivisible logical units of the file to be processed are determined, and the first block parameters are determined based on the indivisible logical units. The second block parameters are determined based on the preset physical storage pages; The file to be processed is divided into blocks based on the first block parameter and the second block parameter to obtain multiple initial file blocks, and the block position of each initial file block in the file to be processed is recorded. Based on the block position of each initial file block, identify whether there is cross-region edge data at the partition edge of each initial file block; If so, then identify the edge data features contained in the cross-regional edge data, determine the associated file blocks based on the edge data features, and determine the edge supplementary data based on the cross-regional edge data; Based on the edge supplement data, the corresponding partition edges in multiple initial file blocks are updated to obtain multiple target file blocks.

[0008] By adopting the above technical solution, indivisible logical units are determined by analyzing the data characteristics of the file to be processed. This facilitates the accurate matching of the block segmentation results with the content attributes of the file to be processed, avoiding the fragmentation of key data caused by traditional fixed block segmentation. During the block processing, referencing preset physical storage pages facilitates the adaptation of blocks to the storage access characteristics of the hardware, avoiding access redundancy caused by blocks being stored across physical pages. By recording the block position of each initial file block in the file to be processed, processing can be directly started based on each block position after the execution of each initial file block. Furthermore, the processing results are returned according to the block position, avoiding the disorder of the order of content due to transmission or processing disconnection. In addition, by analyzing the associated file blocks corresponding to the cross-region edge data, the cross-region edge data can be supplemented, which facilitates the integrity and comprehensiveness of the cross-region edge data and also improves the accuracy of subsequent return.

[0009] In one possible implementation, determining the edge supplementary data based on the cross-regional edge data includes: Based on the edge data characteristics of the cross-regional edge data and the preset data score mapping relationship, the edge data score corresponding to the cross-regional edge data is determined; Based on a preset logical structure tree and the edge data features, the impact file blocks of the cross-regional edge data are determined, the impact data features of each impact file block are identified, and the impact data score corresponding to the cross-regional edge data is determined according to the impact number and impact data features of the impact file blocks. The amount of supplementary data is determined based on the edge data score and the impact data score; The edge supplementary data is determined based on the edge data characteristics of the cross-regional edge data, the associated file blocks corresponding to the cross-regional edge data, and the amount of supplementary data.

[0010] By adopting the above technical solution, the edge data characteristics of cross-regional edge data are analyzed and quantified into edge data scores, which makes it easier to intuitively understand the importance of the cross-regional edge data itself. Then, the influence file blocks with related cross-regional edge data are analyzed through a preset logical structure tree, and the importance of the corresponding influence file blocks is determined. Finally, the amount of data to be supplemented is determined by combining the importance of the cross-regional edge data itself and the importance of the corresponding influence file blocks. This helps to improve the fit between the supplemented content and the actual importance, and avoids insufficient or excessive supplementation.

[0011] In one possible implementation, when executing the corresponding block execution instructions for each object file block, the following is also included: The data security monitoring frequency corresponding to each target file block is determined based on the block characteristics of each target file block, and data security monitoring is performed on each target file block based on each data security monitoring frequency. When abnormal file segments are detected, repair information is generated based on the abnormal file segments, and the repair information is fed back.

[0012] By adopting the above technical solution and analyzing the segmentation characteristics of the target file blocks, it is easy to customize the data security monitoring frequency for different blocks, thereby making it easier to accurately allocate security protection resources to high-risk file blocks. In addition, security monitoring that matches the monitoring frequency is performed for each target file block, which makes it easier to capture potential risks in the corresponding file block processing process in real time, thereby helping to avoid business interruptions caused by security risks.

[0013] In one possible implementation, the method further includes: Identify the block data characteristics corresponding to the abnormal file blocks, and determine the data importance level corresponding to the abnormal file blocks based on the block data characteristics; Based on the characteristics of the segmented data, the abnormal associated file segments corresponding to the abnormal file segments are determined, and the associated execution progress of each abnormal associated file segment at the observation time is determined according to the execution log information. Based on the associated execution progress of each abnormal associated file block at the observation time, determine the target associated execution impact value corresponding to the abnormal file block; The block impact value corresponding to the abnormal file block is determined based on the data importance level corresponding to the abnormal file block, the target association execution impact value, and the number of abnormal associations in the abnormal associated file block. When the block impact value is higher than the preset impact threshold, a stop execution instruction is generated.

[0014] By adopting the above technical solution, the block data characteristics of abnormal file segments are analyzed, making it easier to classify their importance. This allows subsequent processing resources to be tilted towards higher-importance anomalies, avoiding resource misallocation. By locating the anomaly-related file segments through block data characteristics and extracting their related execution progress at the observation time by combining execution log information, it is easier to grasp the spread status of the anomaly content. This dual assessment of the scope of association and execution progress provides a dynamic dimension for quantifying the impact of anomalies. Finally, multi-dimensional quantitative assessment transforms the decision of whether to stop execution from experience-driven to data-driven, improving the adaptability between intervention and actual situation. Ultimately, it provides an intelligent anomaly response mechanism for parallel processing of large files that can both promptly curb the spread of risks and ensure processing continuity, thus improving the stability and reliability of the system in complex environments.

[0015] In one possible implementation, determining the target associated execution impact value corresponding to the abnormal file block based on the associated execution progress of each abnormal associated file block at the observation time includes: The abnormal execution progress of the abnormal file block at the observation time is determined based on the execution log information. Based on the abnormal execution progress and the associated execution progress of the corresponding abnormal associated file blocks, determine the associated execution impact weight corresponding to each abnormal associated file block; The dependency data and degree of dependency between the abnormal file block and each abnormal associated file block are determined according to the preset dependency relationship graph, and the dependency execution progress of the dependent blocks in each abnormal associated file block is determined according to the execution log information. The dependent block is a part of the file block containing the corresponding dependency data in the abnormal associated file block. Based on the dependency level and dependency execution progress of each abnormal associated file block, determine the associated execution impact value of each abnormal associated file block; The target associated execution impact value of the abnormal file block is determined based on the associated execution impact weight and associated execution impact value of each associated abnormal file block corresponding to the abnormal file block.

[0016] By adopting the above technical solution, the associated execution progress of abnormal files is analyzed and the associated execution progress of corresponding abnormal file blocks is determined based on the comparison results. This facilitates the improvement of the fit between the impact on each abnormal file block and the actual dependency situation. Through in-depth mining of the preset dependency graph, the actual dependency degree between abnormal file blocks and abnormal file blocks can be further determined. By analyzing the specific execution progress of the dependent data, the lower limit of the actual impact of the abnormal situation at the current stage can be analyzed. Finally, by comprehensively analyzing the associated execution impact weight and associated execution impact value of each abnormal file block corresponding to the abnormal file block, the accuracy of determining the target associated execution impact value of the abnormal file block can be improved.

[0017] Secondly, this application provides a parallel processing system, which adopts the following technical solution: A parallel processing system comprising: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: execute the above-described file chunking parallel processing method based on dynamic thresholds.

[0018] Thirdly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium includes: a computer program stored thereon that can be loaded by a processor and execute the above-described file block parallel processing method based on dynamic thresholds.

[0019] Fourthly, this application provides a computer program product, which adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the above-described file block parallel processing method based on dynamic thresholds.

[0020] In summary, this application includes at least one of the following beneficial technical effects: 1. By analyzing the processing parameters and data characteristics of the file to be processed, the block segmentation judgment conditions of the file to be processed are determined, which facilitates the improvement of the effectiveness of the block segmentation processing results. Block segmentation processing is performed based on data characteristics and preset physical storage pages, which facilitates the improvement of the accuracy of the block segmentation processing results. By analyzing the block segmentation characteristics of each target block and the corresponding adaptive parameters, it is convenient to generate differentiated block segmentation execution instructions for different target file blocks. While meeting diverse processing needs, it is also convenient to improve processing efficiency. By recording the block position of the target file blocks in the file to be processed, it is convenient to ensure that all target file blocks can be accurately reassembled in their original order after completing their respective execution tasks. This helps to avoid file corruption caused by disordered order in traditional block segmentation processing, thereby improving the accuracy in the processing of large data packages. 2. By analyzing the block data characteristics of abnormal files, it is easier to classify their importance, thereby allowing subsequent processing resources to be tilted towards higher-importance anomalies and avoiding resource misallocation. By locating the file blocks associated with the anomalies through block data characteristics and extracting their associated execution progress at the observation time by combining execution log information, it is easier to grasp the spread status of the abnormal content. This dual assessment of the scope of association and execution progress provides a dynamic basis for quantifying the impact of anomalies. Finally, through multi-dimensional quantitative assessment, the decision on whether to stop execution is shifted from experience-driven to data-driven, which facilitates the adaptation between the intervention intensity and the actual situation. Finally, it provides an intelligent anomaly response mechanism for parallel processing of large files that can both promptly curb the spread of risks and ensure the continuity of processing, thereby improving the stability and reliability of the system in complex environments. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a file block parallel processing method based on dynamic thresholds in an embodiment of this application. Figure 2 This is a schematic diagram of a structure for determining a stop execution instruction in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a parallel processing system according to an embodiment of this application. Detailed Implementation

[0022] The following is in conjunction with the appendix Figures 1 to 3 This application will be described in further detail.

[0023] After reading this specification, those skilled in the art may make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0026] Specifically, this application provides a file chunking parallel processing method based on dynamic thresholds, executed by a parallel processing system. This parallel processing system can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0027] refer to Figure 1 , Figure 1 This is a flowchart illustrating a file block parallel processing method based on dynamic thresholds in an embodiment of this application. The method includes steps S110-S140, wherein: Step S110: Identify the processing parameters and data characteristics of the file to be processed at the current time, and determine the block processing threshold of the file to be processed based on the processing parameters and data characteristics.

[0028] Specifically, the files to be processed are those that require transmission, storage, or compression. The file type of the files to be processed is not specifically limited in this embodiment. Processing parameters characterize the current working state of the parallel processing system. These parameters include, but are not limited to, load, available memory capacity, and network transmission rate. They can be collected by parameter collectors pre-deployed on each node and uploaded to the parallel processing system. Data features corresponding to the files to be processed can be identified from the files based on a preset feature recognition algorithm. These data features mainly characterize the file type, including but not limited to video, audio, documents, images, compressed files, and program files.

[0029] Since the processing parameters at the current moment may affect the file processing process, and different processing behaviors may occur when processing files with different data characteristics under the same processing parameters, this application does not use fixed block division judgment conditions. Instead, it determines the applicable block division judgment conditions, i.e., block division thresholds, based on the data characteristics of the file to be processed and the processing parameters of the parallel processing system at the current moment. The data volume of the file to be processed is compared with the corresponding block division threshold to determine whether the file to be processed needs to be block-divided. Specifically, the block division thresholds corresponding to the processing parameters and data characteristics can be determined based on a preset block division mapping relationship. The preset block division mapping relationship is the correspondence between the parameter combination of processing parameters and data characteristics and the block division threshold. The specific content of this mapping relationship is not specifically limited in this embodiment of the application, but can be determined by relevant personnel based on historical experimental data and then uploaded to the parallel processing system.

[0030] Step S120: When the amount of data to be processed in the file to be processed is greater than the block processing threshold, the file to be processed is divided into blocks based on data characteristics and preset physical storage pages to obtain multiple target file blocks.

[0031] Specifically, the amount of data to be processed in the file to be processed can be identified based on a preset feature recognition algorithm, and the amount of data to be processed can be compared with the determined block processing threshold. When the amount of data to be processed is greater than the block processing threshold, it indicates that the parallel processing system may experience stuttering, packet loss, delays, etc. when processing the file as a whole. In order to ensure the integrity and security of the file to be processed during the processing, the file to be processed can be divided into multiple target file blocks. By processing each target file block in parallel, it is convenient to avoid the overall processing bottleneck of the parallel processing system while taking into account processing efficiency, integrity and security, and to optimize the entire process of large file processing.

[0032] The preset physical storage page is the lowest-level operation unit of the system. Its core function is to match the block partitioning strategy with the system's physical read / write rules, avoiding resource waste or performance loss caused by incompatibility between block size and storage page. That is, the block partitioning process must use the preset physical storage page as the smallest unit. Even if only 1 byte of data needs to be read, the entire storage page must be loaded. If the target file block size is not aligned with the storage page, read / write operations for a single target file block may span multiple pages, significantly increasing I / O overhead. The preset physical storage page size can be 4KB, 8KB, or 16KB. The specific preset physical storage page size is not specifically limited in this embodiment and can be set by relevant personnel according to actual conditions. Furthermore, since data features are the embodiment of the inherent attributes of the file to be processed, the content structure and encoding format of the file to be processed can be understood based on the data features. Block processing based on data features helps to avoid destroying the logical integrity of the file to be processed during the block processing. If the block is only divided according to the preset physical storage pages and the data features are ignored, the block of the target file may not be able to be merged or parsed and used normally. Therefore, when processing the file to be processed into blocks, the logical block candidate range can be defined first based on the data features, and then the physical alignment standard can be determined based on the preset physical storage pages. After the file to be processed is divided into blocks based on the logical block candidate range and the physical alignment standard, multiple target file blocks are obtained.

[0033] Furthermore, to improve the accuracy of determining target file blocks, the method provided in this application, when dividing the file to be processed into multiple target file blocks based on data characteristics and preset physical storage pages, is specifically used for: Based on data characteristics, indivisible logical units of the file to be processed are determined, and first block parameters are determined based on these indivisible logical units. Second block parameters are determined based on preset physical storage pages. The file to be processed is divided into blocks based on the first and second block parameters to obtain multiple initial file blocks, and the block position of each initial file block in the file to be processed is recorded. Based on the block position of each initial file block, it is identified whether there is cross-regional edge data at the partition edges of each initial file block. If so, the edge data features contained in the cross-regional edge data are identified, and associated file blocks are determined based on the edge data features. Edge supplementary data is determined based on the cross-regional edge data. The corresponding partition edges in the multiple initial file blocks are updated based on the edge supplementary data to obtain multiple target file blocks.

[0034] Specifically, different data features correspond to different indivisible logical units. The indivisible logical units corresponding to the file to be processed can be determined based on a preset logical unit mapping relationship. This preset logical unit mapping relationship is the correspondence between data features and indivisible logical units. For example, when the data feature is video, the corresponding indivisible logical unit can be one keyframe + subsequent associated P / B frames; when the data feature is a document, the corresponding indivisible unit can be a single page of content. The specific content of the preset logical unit mapping relationship is not specifically limited in this embodiment. After determining the indivisible logical unit corresponding to the data feature, this indivisible logical unit can be determined as the first block parameter of the file to be processed. The first block parameter is used to constrain the block from destroying the logical unit. After obtaining the preset physical storage page through the system interface, the physical page alignment granularity and storage page address rules can be determined based on the preset physical storage page. The physical page alignment granularity and storage page address rules are then determined as the second block parameter of the file to be processed. The second block parameter is used to constrain the adaptability of the block to the physical storage.

[0035] When processing the file to be processed by dividing it into blocks based on the first and second block parameters, these parameters can be used as constraints. The specific block division process is not limited in this embodiment. For example, initial block division can be performed first based on the first block parameter, and then the initial block division result can be optimized based on the second block parameter, as long as the multiple initial file blocks obtained after block division satisfy the constraints corresponding to the first and second block parameters. Each initial file block has a unique block position within the file to be processed. The block position can be marked using a number or code; the specific form is not limited in this embodiment.

[0036] Cross-region edge data refers to data that has a logical connection at the boundary of the initial file block but has been split into different initial file blocks, such as table rows across blocks or continuous image pixel blocks. The specific form of cross-region edge data is not specifically limited in this embodiment. When identifying whether the initial file block contains cross-region edge data, the end offset of each initial file block can be identified first according to a preset feature recognition algorithm, that is, the partition edge of the initial file block. Then, it is checked whether there are logical connection features in the data within a preset range before and after each partition edge. If so, it is determined that the corresponding initial file block contains cross-region edge data. The initial file block can contain one cross-region edge data or two cross-region edge data. That is, data with logical connection to other initial file partitions is detected within the preset range before and after the partition edges of the initial file block. Cross-region edge data is the data corresponding to the partition edge.

[0037] When cross-region edge data is detected in an initial file block, edge data features can be identified from the cross-region edge data based on a preset feature recognition algorithm. Based on these edge data features, related file blocks logically associated with the edge data features can be determined from other initial file blocks. Cross-region edge data may correspond to one or more related file blocks; the specific number is not specifically limited in this embodiment. Since there is a logical relationship between the data within the related file blocks and the cross-region edge data, some data can be filtered or extracted from the related file blocks as supplementary edge data for the cross-region edge data. This completes the data content at the corresponding partition edge and updates the corresponding initial file block. Based on the above, all initial file blocks containing cross-region edge data can be updated to obtain the target file block corresponding to each initial file block.

[0038] Step S130: Identify the block characteristics of each target file block, generate block execution instructions corresponding to each target file block based on the block characteristics and adaptive parameters, and obtain the execution block corresponding to each target file block. The adaptive parameters are determined based on the processing parameters at the current time. The execution block is the execution result obtained after the target file block is processed in parallel according to the corresponding block execution instructions.

[0039] Specifically, the block characteristics of each target file block can be identified based on a preset feature recognition algorithm. These block characteristics may include, but are not limited to, block size, block density, and block format compatibility. The specific details can be set by the user according to actual conditions and are not limited in this embodiment. Adaptive parameters can be determined by the processing parameters at the current moment. For example, initially, 2GB of memory is allocated to each target file block for data caching, but this is reduced to 1GB when the load exceeds a threshold; or the CPU usage time of a single target file block is limited to avoid a single block occupying core resources for a long time, causing other tasks to queue and become congested. Embedding adaptive parameters in the execution instructions allows the strategy to be adjusted based on real-time performance feedback during block processing. The block execution instructions determined based on the block characteristics and adaptive parameters of each target file block can independently execute the processing tasks of the corresponding target file block. The execution result of the target file block is the result obtained after the target file block completes its corresponding processing task.

[0040] Step S140: After all target file blocks have completed their respective parallel processing, each execution block is rolled back according to its position in the file to be processed, so as to obtain the execution file corresponding to the file to be transmitted.

[0041] Specifically, after all target file blocks have been processed in parallel, a four-step fallback process—block location, data verification, sequential reassembly, and integrity verification—can combine the scattered execution blocks into a complete executable file. The core is to ensure that the reassembled executable file maintains the same logical structure and data content as the original file to be processed. First, all execution blocks are sorted according to their corresponding block positions to obtain an initial sorting queue. Since parallel processing may cause data corruption within some execution blocks due to node failures or network fluctuations, verification and filtering are required after obtaining the initial sorting queue to prevent data errors in the final executable file. Verification and filtering methods can include checksum matching, data length matching, and logical unit verification, etc. The specific verification and filtering methods are not specifically limited in this embodiment. After verification and filtering, the execution blocks in the initial sorting queue can be concatenated to form a continuous executable file.

[0042] In this embodiment of the application, by analyzing the processing parameters and data characteristics of the file to be processed, the block judgment conditions of the file to be processed are determined, which facilitates the improvement of the effectiveness of the block processing results. Block processing is performed based on data characteristics and preset physical storage pages, which facilitates the improvement of the accuracy of the block processing results. By analyzing the block characteristics of each target block and the corresponding adaptive parameters, it is convenient to generate differentiated block execution instructions for different target file blocks. While meeting diverse processing needs, it is also convenient to improve processing efficiency. By recording the block position of the target file blocks in the file to be processed, it is convenient to ensure that all target file blocks can be accurately reassembled in the original order after completing their respective execution tasks. This helps to avoid file corruption caused by disordered order in traditional block processing, thereby improving the accuracy in the processing of large data packages.

[0043] Furthermore, to avoid insufficient or excessive supplementation when determining edge supplementation data, edge supplementation data is determined based on cross-regional edge data, which may specifically include: Based on the edge data characteristics of cross-regional edge data and the preset data score mapping relationship, the edge data score corresponding to the cross-regional edge data is determined; based on the preset logical structure tree and edge data characteristics, the impact file blocks of the cross-regional edge data are determined, the impact data characteristics of each impact file block are identified, and the impact data score corresponding to the cross-regional edge data is determined according to the impact quantity and impact data characteristics of the impact file blocks; the amount of supplementary data is determined based on the edge data score and the impact data score; and edge supplementary data is determined based on the edge data characteristics of the cross-regional edge data, the associated file blocks corresponding to the cross-regional edge data, and the amount of supplementary data.

[0044] Specifically, different cross-regional edge data features correspond to different levels of data importance. Based on the preset data score mapping relationship, the importance of the corresponding cross-regional edge data can be evaluated. In this application embodiment, the edge data score can be used to intuitively represent the importance of cross-regional edge data. The preset data score mapping relationship is the correspondence between edge data features and edge data scores. The specific content is not specifically limited in this application embodiment.

[0045] The preset logical structure tree is a hierarchical tree model describing the logical dependencies between various target file blocks of the file to be processed. It can be determined by relevant technical personnel based on the specific content of the file to be processed and uploaded to the parallel processing system in advance. By traversing the preset logical structure tree based on edge data features, the influencing file blocks that are logically related to the cross-regional edge data can be identified. After identifying all the influencing file blocks, the number of influences of each influencing file block can be recorded, and the influence data features of each influencing file block can be identified based on a preset feature recognition algorithm. Different influence numbers and influence data features correspond to different influence data scores. The influence data score corresponding to the cross-regional edge data can be determined based on a preset influence mapping relationship. The preset influence mapping relationship is the correspondence between the parameter combination of influence number and influence data features and the influence data score. The specific content is not specifically limited in this embodiment of the application.

[0046] After determining the edge data scores and impact data scores, the amount of supplementary data corresponding to the cross-regional edge data can be determined based on a preset supplementary mapping relationship. This preset supplementary mapping relationship corresponds to the combination of edge data scores and impact data scores with the amount of supplementary data; the higher the edge data score and / or the impact data score, the higher the corresponding amount of supplementary data. When determining edge supplementary data based on cross-regional edge data, the associated file blocks corresponding to the cross-regional edge data, and the amount of supplementary data, the scale boundary of the edge supplementary data can be clarified first based on the amount of supplementary data. At this point, it is also necessary to combine the edge data characteristics of the cross-regional edge data to decompose the specific supplementary content direction to avoid insufficient or excessive redundancy. The associated file blocks are the providers of edge supplementary data; that is, it is necessary to determine the edge supplementary data that has a logical relationship with the cross-regional edge data from the content contained in the associated file blocks based on the edge data characteristics and the amount of supplementary data. The finally generated supplementary data can not only solve the problem of fragmented block boundaries but also enable the associated file blocks to be used efficiently in parallel processing, ensuring the integrity and efficiency of the overall file processing.

[0047] Furthermore, to facilitate avoiding business interruptions due to security risks, when executing the corresponding block execution instructions for each target file block, it may also include: The data security monitoring frequency for each target file block is determined based on its block characteristics, and data security monitoring is performed on each target file block based on each data security monitoring frequency. When an abnormal file block is detected, repair information is generated based on the abnormal file block, and the repair information is fed back.

[0048] Specifically, different block characteristics can be used to characterize the storage data of corresponding target file blocks. Based on the block characteristics of each target file block, the required security level for each target file block can be estimated. That is, it is necessary to assign appropriate data security monitoring frequencies to different target file blocks by analyzing the block characteristics. The higher the data security monitoring frequency, the higher the required security level for the corresponding target file block. The data security monitoring frequency corresponding to each target file block can be determined based on a preset monitoring frequency mapping relationship. The preset monitoring frequency mapping relationship is the correspondence between block characteristics and data security monitoring frequencies. The specific content is not specifically limited in this application embodiment. After determining the data security monitoring frequency, security monitoring can be performed on each target file block during the execution of its respective parallel tasks based on the data security monitoring frequency. When abnormal execution characteristics are detected, it can be determined that there is an abnormal file block. At this time, repair information needs to be generated quickly and a response mechanism needs to be triggered. The abnormal execution characteristics may include, but are not limited to, tampering anomalies, damage anomalies, and leakage anomalies. Performing security monitoring matching its monitoring frequency for each target file block facilitates the real-time capture of potential risks in the processing of the corresponding file block.

[0049] Furthermore, to improve stability and reliability in complex environments, the method provided in this application may further include steps S210-250, such as... Figure 2 As shown, where: Step S210: Identify the block data characteristics corresponding to the abnormal file blocks, and determine the data importance level corresponding to the abnormal file blocks based on the block data characteristics.

[0050] Specifically, the data importance level of the abnormal file block can be determined first based on a preset importance level mapping relationship. The preset importance level mapping relationship is the correspondence between the block data characteristics and the data importance level. Based on the preset importance level mapping relationship, the data importance level corresponding to the abnormal file block can be determined. The specific content of the preset importance level mapping relationship is not specifically limited in this application embodiment.

[0051] Step S220: Based on the characteristics of the segmented data, determine the abnormal associated file segments corresponding to the abnormal file segments, and determine the associated execution progress of each abnormal associated file segment at the observation time according to the execution log information.

[0052] Specifically, anomaly-related file blocks refer to other target file blocks that have direct or indirect data dependencies or logical connections with the anomaly file block. They can be located using features such as association identifiers, dependency characteristics, and structural location characteristics in the block data. For example, when using structural location characteristics to locate anomaly-related file blocks, the file blocks corresponding to the parent, child, or adjacent nodes of the anomaly file block can be traversed in a preset logical structure tree based on the block data characteristics, and these can be considered as anomaly-related file blocks. The execution log information contains the execution progress of each target file block. Due to differences in the attributes of different target file blocks and differences in the resource allocation of the execution environment, different target file blocks have different execution times and priorities in the execution process. The associated execution progress is the execution progress of the anomaly-related file block.

[0053] Step S230: Determine the target associated execution impact value of each abnormal file block based on the associated execution progress at the observation time.

[0054] Specifically, the observation time is the moment when the abnormal file block is identified. The execution progress of the abnormal associated file block directly reflects the depth of its impact on the abnormal associated file block. For example, if the execution progress of an abnormal associated file block reaches 90% at the observation time, it indicates that the abnormality of the abnormal file block may only intervene in the later stage of its execution, affecting only the remaining 10% of the tasks, with minimal damage to the overall execution result of that abnormal associated file block. Conversely, if the execution progress of another associated file block is only 10%, it indicates that the abnormality of the abnormal file block intervened in the early stage of its execution, resulting in almost no progress in its core tasks, and significant damage to the execution result of that abnormal associated file block. The target associated execution impact value corresponding to the abnormal file block can be determined by summing the associated execution progress of all abnormal associated file blocks. Furthermore, to improve the accuracy of determining the target associated execution impact value, the determination of the target associated execution impact value based on the associated execution progress of each abnormal associated file block at the observation time can specifically include: The abnormal execution progress of the abnormal file block at the observation time is determined based on the execution log information. Based on the abnormal execution progress and the associated execution progress of the corresponding abnormal file blocks, the associated execution impact weight of each abnormal file block is determined. The dependency data and degree of dependency between the abnormal file block and each abnormal associated file block are determined based on the preset dependency graph. The dependency execution progress of the dependent blocks in each abnormal associated file block is determined based on the execution log information; the dependent blocks are the file blocks in the abnormal associated file block that contain the corresponding dependency data. The associated execution impact value of each abnormal associated file block is determined based on the degree of dependency and the dependency execution progress. Finally, the target associated execution impact value of the abnormal file block is determined based on the associated execution impact weight and associated execution impact value of each abnormal associated file block.

[0055] Specifically, the abnormal execution progress of the abnormal file block at the observation time is determined from the execution log information. This abnormal execution progress is then compared with the associated execution progress of each associated abnormal file block to determine the progress difference between the associated and abnormal execution progress. Based on a preset execution influence weight mapping relationship, the associated execution influence weight corresponding to each progress difference is determined. The progress difference includes negative and positive differences. For example, if the abnormal execution progress is 90% and the associated execution progress is 10%, the progress difference between the two is 80%; if the abnormal execution progress is 10% and the associated execution progress is 90%, the progress difference between the two is -80%. The larger the absolute value of the progress difference, the greater the corresponding execution influence weight. The specific content of the preset execution influence weight mapping relationship is not specifically limited in this embodiment and can be determined by relevant personnel based on historical experimental data and then uploaded to the parallel processing system.

[0056] A pre-defined dependency graph is a structured model that records the data interaction rules between target file blocks. Based on this graph, the degree of dependency between abnormal file blocks and their associated blocks can be determined; higher interaction frequency corresponds to a higher degree of dependency. Dependency data refers to the call data involved in the interaction. To facilitate interaction, call data is typically stored separately in a portion of the target file block, i.e., the dependency block. The execution progress of dependent blocks in each associated file block can be identified from the execution log information. Finally, based on a pre-defined execution impact mapping relationship, the associated execution impact value for each associated file block is determined. This pre-defined execution impact mapping relationship is the correspondence between the dependency parameter combination of dependency degree and dependency execution progress and the associated execution impact value. It can be determined by relevant personnel based on historical experimental data and then uploaded to the parallel processing system.

[0057] After determining the execution impact weight and value of each associated file block corresponding to the abnormal file block, the target execution impact value of the abnormal file block can be determined by weighted summation. By analyzing the specific execution progress of dependent data, the lower limit of the actual impact of the abnormal situation at the current stage can be analyzed. Finally, by comprehensively analyzing the execution impact weight and value of each associated file block corresponding to the abnormal file block, the accuracy of determining the target execution impact value of the abnormal file block can be improved.

[0058] Step S240: Determine the block impact value corresponding to the abnormal file block based on the data importance level, target association execution impact value, and the number of abnormal associations in the abnormal file block.

[0059] Step S250: When the block impact value is higher than the preset impact threshold, a stop execution instruction is generated.

[0060] Specifically, the data importance level, target association execution impact value, and number of abnormal associations in the abnormal file blocks can be normalized and mapped to the [0,1] interval to eliminate the dimensional differences between the data importance level, target association execution impact value, and number of abnormal associations, thus obtaining the block impact value of the abnormal file blocks. By normalizing the multi-source influencing factors, it is easier to intuitively quantify the abnormal impact caused by the occurrence of abnormal situations in the abnormal file blocks.

[0061] When the block impact value is higher than the preset impact threshold, it indicates that the abnormal file block has a significant impact due to an abnormal situation. At this time, a stop execution command can be generated to pause the execution of the abnormal file block and its associated file blocks to avoid generating too much invalid data. When the block impact value is not higher than the preset impact threshold, it indicates that the abnormal impact of the abnormal file block is within an acceptable and controllable range. At this time, the execution results of the abnormal file block and its associated file blocks can be marked to remind relevant personnel to manually verify the marked execution results.

[0062] In this embodiment of the application, by parsing the block data characteristics of abnormal file blocks, it is easy to classify their importance, thereby facilitating the allocation of subsequent processing resources to high-importance abnormalities and avoiding resource misallocation. By locating abnormal file blocks through block data characteristics and combining them with execution log information to extract their associated execution progress at the observation time, it is easy to grasp the spread status of abnormal content. This dual assessment of the scope of association and execution progress provides a dynamic dimension for quantifying the impact of abnormalities. Finally, through multi-dimensional quantitative assessment, the decision on whether to stop execution is transformed from experience-driven to data-driven, which facilitates the improvement of the adaptability between intervention and actual situation. Finally, it provides an intelligent abnormality response mechanism for parallel processing of large files that can both promptly curb the spread of risks and ensure the continuity of processing, thereby improving the stability and reliability of the system in complex environments.

[0063] This application provides a parallel processing system, such as... Figure 3 As shown, Figure 3 The parallel processing system 300 shown includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the parallel processing system 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one, and the structure of this parallel processing system 300 does not constitute a limitation on the embodiments of this application.

[0064] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0065] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0066] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0067] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0068] Parallel processing systems include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. They can also be servers. Figure 3 The parallel processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0069] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0070] This application provides a computer program product including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0071] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0072] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A file block parallel processing method based on dynamic threshold, characterized in that, include: Identify the processing parameters and data characteristics of the file to be processed at the current time, and determine the block processing threshold of the file to be processed based on the processing parameters and data characteristics; When the amount of data to be processed in the file to be processed is greater than the block processing threshold, the file to be processed is divided into blocks based on the data characteristics and the preset physical storage pages to obtain multiple target file blocks. Identify the block characteristics of each target file block, generate block execution instructions corresponding to each target file block based on the block characteristics and adaptive parameters, and obtain the execution block corresponding to each target file block. The adaptive parameters are determined based on the processing parameters at the current time, and the execution block is the execution result obtained after the target file block is processed in parallel according to the corresponding block execution instructions. Once all target file blocks have completed their respective parallel processing, the execution blocks are rolled back according to their positions in the file to be processed, thus obtaining the execution file corresponding to the file to be transmitted.

2. The file block parallel processing method based on dynamic threshold according to claim 1, characterized in that, The process involves dividing the file to be processed into blocks based on the data characteristics and preset physical storage pages, resulting in multiple target file blocks, including: Based on the data characteristics, the indivisible logical units of the file to be processed are determined, and the first block parameters are determined based on the indivisible logical units. The second block parameters are determined based on the preset physical storage pages; The file to be processed is divided into blocks based on the first block parameter and the second block parameter to obtain multiple initial file blocks, and the block position of each initial file block in the file to be processed is recorded. Based on the block position of each initial file block, identify whether there is cross-region edge data at the partition edge of each initial file block; If so, then identify the edge data features contained in the cross-regional edge data, determine the associated file blocks based on the edge data features, and determine the edge supplementary data based on the cross-regional edge data; Based on the edge supplementation data, the corresponding partition edges in multiple initial file blocks are updated to obtain multiple target file blocks.

3. The file block parallel processing method based on dynamic threshold according to claim 2, characterized in that, The determination of edge supplementary data based on the cross-regional edge data includes: Based on the edge data characteristics of the cross-regional edge data and the preset data score mapping relationship, the edge data score corresponding to the cross-regional edge data is determined; Based on a preset logical structure tree and the edge data features, the impact file blocks of the cross-regional edge data are determined, the impact data features of each impact file block are identified, and the impact data score corresponding to the cross-regional edge data is determined according to the impact number and impact data features of the impact file blocks. The amount of supplementary data is determined based on the edge data score and the impact data score; The edge supplementary data is determined based on the edge data characteristics of the cross-regional edge data, the associated file blocks corresponding to the cross-regional edge data, and the amount of supplementary data.

4. The file block parallel processing method based on dynamic threshold according to claim 2, characterized in that, When executing the corresponding block execution instructions for each target file block, it also includes: The data security monitoring frequency corresponding to each target file block is determined based on the block characteristics of each target file block, and data security monitoring is performed on each target file block based on each data security monitoring frequency. When abnormal file segments are detected, repair information is generated based on the abnormal file segments, and the repair information is fed back.

5. The file block parallel processing method based on dynamic threshold according to claim 4, characterized in that, Also includes: Identify the block data characteristics corresponding to the abnormal file blocks, and determine the data importance level corresponding to the abnormal file blocks based on the block data characteristics; Based on the characteristics of the segmented data, the abnormal associated file segments corresponding to the abnormal file segments are determined, and the associated execution progress of each abnormal associated file segment at the observation time is determined according to the execution log information. Based on the associated execution progress of each abnormal associated file block at the observation time, determine the target associated execution impact value corresponding to the abnormal file block; The block impact value corresponding to the abnormal file block is determined based on the data importance level corresponding to the abnormal file block, the target association execution impact value, and the number of abnormal associations in the abnormal associated file block. When the block impact value is higher than the preset impact threshold, a stop execution instruction is generated.

6. The file block parallel processing method based on dynamic threshold according to claim 5, characterized in that, The step of determining the target associated execution impact value corresponding to the abnormal file block based on the associated execution progress of each abnormal associated file block at the observation time includes: The abnormal execution progress of the abnormal file block at the observation time is determined based on the execution log information. Based on the abnormal execution progress and the associated execution progress of the corresponding abnormal associated file blocks, determine the associated execution impact weight corresponding to each abnormal associated file block; The dependency data and degree of dependency between the abnormal file block and each abnormal associated file block are determined according to the preset dependency relationship graph, and the dependency execution progress of the dependent blocks in each abnormal associated file block is determined according to the execution log information. The dependent block is a part of the file block containing the corresponding dependency data in the abnormal associated file block. Based on the dependency level and dependency execution progress of each abnormal associated file block, determine the associated execution impact value of each abnormal associated file block; The target associated execution impact value of the abnormal file block is determined based on the associated execution impact weight and associated execution impact value of each associated abnormal file block corresponding to the abnormal file block.

7. A parallel processing system, characterized in that, This parallel processing system includes: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: perform a file chunking parallel processing method based on a dynamic threshold as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, include: The computer program is stored and can be loaded by a processor and executed as described in any one of claims 1-6, which is a file block parallel processing method based on dynamic thresholds.

9. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the steps of a file block parallel processing method based on a dynamic threshold, as described in any one of claims 1-6.

Images