Data processing method, device, medium and program product

Abstract

The application provides a data processing method, device, medium and program product, which can be applied to the technical field of data storage. The method comprises the following steps: in response to receiving a data processing instruction, determining a storage space in which to-be-processed data is stored in a memory based on the data processing instruction; dividing the to-be-processed data into a plurality of to-be-processed data blocks arranged in sequence according to storage addresses in the storage space, the data amount of the plurality of to-be-processed data blocks being the same; processing the plurality of to-be-processed data blocks in turn by using a sliding window, and each processing comprises the following steps: selecting at least one to-be-processed data block as a target data set by using the sliding window, the width of the sliding window being an integer multiple of the data amount of the to-be-processed data block; performing multiple overwrites on data in the target data set to clear original data of the target data set; and in the case that there is an unprocessed to-be-processed data block, moving the sliding window according to a preset step length so as to perform the next processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data processing method, device, medium and program product. BACKGROUND

[0002] Data processing is necessary for ensuring data security and optimizing storage resource utilization. Currently, with the explosive growth of data and the complexity of computing environment, existing technologies usually process data in a fixed mode, such as processing the entire data region at one time or clearing data by simple overwriting, to meet basic data processing needs, ensure that data is not accessed arbitrarily after the end of the life cycle, and release storage resources for other tasks.

[0003] In the implementation of the present application, it is found that the related art has at least the following problems. The fixed processing mode often needs to complete data clearing in the entire storage space to be processed before releasing the storage space for executing other data storage tasks at one time. Therefore, it is difficult to release the storage space in time, which affects the utilization rate of the storage space. SUMMARY

[0004] In view of the above problems, the present application provides a data processing method, device, medium and program product.

[0005] According to a first aspect of the present application, a data processing method is provided, comprising: in response to receiving a data processing instruction, determining a storage space in which data to be processed is stored in the storage, based on the data processing instruction; dividing the data to be processed into a plurality of data blocks to be processed arranged in sequence according to the storage address in the storage space, the data amount of the plurality of data blocks to be processed being the same; using a sliding window to process the plurality of data blocks to be processed in turn, wherein each processing includes: using the sliding window to select at least one data block to be processed as a target data set, wherein the width of the sliding window is an integer multiple of the data amount of the data block to be processed; performing multiple overwrites on the data in the target data set to clear the original data of the target data set; in the case where there is unprocessed data block to be processed, moving the sliding window by a preset step to execute the next processing, wherein the preset step is related to the width of the sliding window.

[0006] The second aspect of the present application provides a data processing apparatus, comprising: a space determining module configured to determine a storage space in which data to be processed is stored in a memory based on a data processing instruction in response to receiving the data processing instruction; a data dividing module configured to divide the data to be processed into a plurality of data blocks arranged in sequence according to storage addresses in the storage space, the plurality of data blocks having the same data amount; and a data block processing module configured to process the plurality of data blocks in sequence by using a sliding window, wherein the data block processing module comprises: a data block selecting submodule configured to select at least one data block to be processed as a target data set by using the sliding window, wherein a width of the sliding window is an integer multiple of the data amount of the data block to be processed; a data overwriting submodule configured to overwrite data in the target data set multiple times to clear original data of the target data set; and a window sliding submodule configured to move the sliding window according to a preset step size to perform the next processing in the case that there is an unprocessed data block to be processed, wherein the preset step size is related to the width of the sliding window.

[0007] The third aspect of the present application provides an electronic device, comprising: one or more processors; and a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0008] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the method.

[0009] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the method. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0011] Figure 1 An application scenario diagram of the data processing method, device, medium and program product according to the embodiments of the present application is shown.

[0012] Figure 2 A flowchart of the data processing method according to the embodiments of the present application is shown.

[0013] Figure 3 A schematic diagram of dividing the data to be processed into data blocks and a sliding window according to the data processing method according to the embodiments of the present application is shown.

[0014] Figure 4A schematic diagram showing data processing according to the data processing method of an embodiment of the present application is shown.

[0015] Figure 5 A flowchart showing a data processing method according to another embodiment of the present application is shown.

[0016] Figure 6 A block diagram showing the structure of a data processing apparatus according to an embodiment of the present application is shown.

[0017] Figure 7 A block diagram showing an electronic device suitable for implementing the data processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, the drawings are designed for a better understanding, and are not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art, that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, means the inclusion of the stated features, steps, operations, and / or components but not to the exclusion of one or more other features, steps, operations, or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the understanding of those terms by those in the relevant art, and not in an overly idealized or overly formal sense.

[0021] In instances where expressions such as "at least one of A, B, and C, etc." are used, in general, it should be interpreted that the meaning is in accordance with the understanding of the expression by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0022] Embodiments of the present application provide a data processing method, comprising: in response to receiving a data processing instruction, determining, based on the data processing instruction, a storage space in a memory in which to-be-processed data is stored; dividing the to-be-processed data into a plurality of to-be-processed data blocks arranged in sequence according to storage addresses in the storage space, the plurality of to-be-processed data blocks having the same data amount; and processing the plurality of to-be-processed data blocks in sequence by using a sliding window, wherein each processing comprises: selecting at least one to-be-processed data block as a target data set by using the sliding window, wherein the width of the sliding window is an integer multiple of the data amount of the to-be-processed data block; and performing multiple overwrites on data in the target data set to clear original data of the target data set; and in the case that there is an unprocessed to-be-processed data block, moving the sliding window by a preset step length so as to perform the next processing, wherein the preset step length is related to the width of the sliding window.

[0023] Figure 1 An application scenario diagram of the data processing method, device, medium and program product according to embodiments of the present application is shown.

[0024] As shown in Figure 1 application scenario 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0025] A user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0026] The first terminal device 101, the second terminal device 102, the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0027] The server 105 can be a server providing various services, such as a background management server providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back a processing result (such as a webpage, information, or data, or the like, obtained or generated according to a user request) to the terminal device.

[0028] It should be noted that the data processing method provided in the embodiments of the present application can generally be executed by the server 105. Correspondingly, the data processing apparatus provided in the embodiments of the present application can generally be arranged in the server 105. The data processing method provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the data processing apparatus provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0029] It should be understood that Figure 1 The number of the first terminal device 101, the second terminal device 102, the third terminal device 103, the network, and the server in the above

[0030] The data processing method of the embodiments of the present application will be described in detail below based on the scenario described above. Figure 1 Figures 2-5 The data processing method of the embodiments of the present application will be described in detail below based on the scenario described above.

[0031] Figure 2 A flowchart of the data processing method according to the embodiments of the present application is shown.

[0032] As shown in Figure 2 The data processing method of the embodiments includes the following operations.

[0033] In operation S210, in response to receiving a data processing instruction, a storage space in which the to-be-processed data is stored in the memory is determined based on the data processing instruction.

[0034] In operation S220, the to-be-processed data is divided into a plurality of to-be-processed data blocks arranged in sequence according to storage addresses in the storage space, and the data amount of the plurality of to-be-processed data blocks is the same.

[0035] In operation S230, the plurality of to-be-processed data blocks are processed in sequence by using a sliding window.​

[0036] The processing in any one of operation S230 includes operations S231-S233:

[0037] In operation S231, at least one to-be-processed data block is selected as a target data set using a sliding window, wherein the width of the sliding window is an integer multiple of the data amount of the to-be-processed data block.

[0038] In operation S232, the data in the target data set is overwritten multiple times to clear the original data in the target data set.

[0039] In operation S233, when there is a to-be-processed data block that has not been processed, the sliding window is moved by a preset step to perform the next processing, wherein the preset step is related to the width of the sliding window.

[0040] According to an embodiment of the present application, the data processing instruction can be used to instruct to clear the to-be-processed data in the memory and release the storage space used to store the to-be-processed data, wherein the memory identifier, the storage space identifier, etc. can be included in the data storage instruction.

[0041] According to an embodiment of the present application, after receiving the data processing instruction, the data processing instruction is parsed to determine the memory identifier and the storage space identifier included therein, and after determining the memory that needs to be processed according to the memory identifier, the storage space storing the to-be-processed data is located from the memory according to the storage space identifier.

[0042] According to an embodiment of the present application, the to-be-processed data is divided according to the storage addresses of the to-be-processed data in the storage space, so that the data amount of each to-be-processed data block is the same, and the order of the plurality of to-be-processed data blocks obtained after the division is determined according to the storage addresses of the data blocks in the storage space.

[0043] According to an embodiment of the present application, when the plurality of to-be-processed data blocks are processed using the sliding window, the first number of to-be-processed data blocks that can be covered by the sliding window at a time is determined according to the width of the sliding window and the data amount of each to-be-processed data block. According to the order of the plurality of to-be-processed data blocks, the first number of to-be-processed data blocks covered by the sliding window are selected as the target data set that needs to be processed in the current batch.

[0044] According to an embodiment of the present application, the plurality of bits of data stored in the target data set are overwritten multiple times respectively, so that after the target data set is overwritten, the original data in the storage space used to save the target data set has been covered, thereby completing the clearing and clearing of the original data in the target data set.

[0045] According to an embodiment of the present application, after the processing of the target data set of the current batch is completed, it is determined whether there are still unprocessed data blocks to be processed. In the case where there are unprocessed data blocks to be processed, a preset step size is determined according to the width of the sliding window, and the sliding window is moved according to the preset step size. The first number of data blocks to be processed covered by the sliding window are taken as the target data set of the next batch to be processed, so as to perform the next processing on the target data set of the next batch to be processed.

[0046] According to an embodiment of the present application, by dividing the data to be processed into data blocks to be processed with the same amount of data, the target data set is selected by the sliding window in turn, and the original data in the target data set is cleared by multiple overwriting, so as to realize the ordered and complete processing of the data to be processed in the memory. The sliding window mechanism combined with the preset step size movement can ensure the continuity and integrity of data processing, avoid data omission or repeated processing, and the equal block design makes the processing process more regular, thereby improving the efficiency and reliability of data clearing.

[0047] Figure 3 A schematic diagram of the data processing method according to an embodiment of the present application is shown.

[0048] As shown in Figure 3 The storage space 300 of the memory is divided into three storage volumes, i.e., a first storage volume 310, a second storage volume 320 and a third storage volume 330. According to the data processing instruction, the data to be processed to be cleared is determined from the storage space 300, and the data to be processed does not need to be stored continuously in the memory. In this example, the data to be processed is stored in the first storage volume 310, the second storage volume 320 and the third storage volume 330. According to the storage addresses of the respective storage spaces corresponding to the data to be processed, the data to be processed can be divided into a plurality of data blocks to be processed. The plurality of data blocks to be processed can be placed in a target data set, obtaining a target data set 340 to be processed.

[0049] In the example, the width of the sliding window is three times the size of the data block to be processed, that is, the sliding window can cover three data blocks to be processed each time it slides to a position, and when the target data set to be processed is located at the first position 351, the first target data set 361 can be determined according to the three data blocks to be processed covered thereby. After the processing of the first target data set 361 is completed, it is determined that there are still data blocks to be processed in the target data set to be processed 340, and therefore the sliding window can be moved according to the preset step size, to the second position 352, and the second target data set 362 can be determined according to the three data blocks to be processed covered thereby at the second position 352, and the processing of the second target data set 362 is continued. When it is determined that there are no data blocks to be processed in the target data set to be processed 340, it can be determined that the processing of the data to be processed has been completed.

[0050] According to an embodiment of the present application, the data processing method further comprises periodically acquiring a load parameter of a computing device associated with the memory, and comparing the load parameter of the computing device with a reference parameter of the computing device to obtain a comparison result; and adjusting the sliding window according to the comparison result, so that the load parameter of the computing device is consistent with the reference parameter of the computing device when the processing is performed using the adjusted sliding window.

[0051] According to an embodiment of the present application, the computing device associated with the memory can include a server where the memory is located or a server connected to the memory, and the load parameter can include a Central Processing Unit (CPU) utilization rate, a memory utilization rate, and other parameters for measuring the load of the server.

[0052] According to an embodiment of the present application, the reference parameter of the computing device can include a CPU reference utilization rate and a memory reference utilization rate and other parameters when the computing device is in a load-balanced state.

[0053] According to an embodiment of the present application, the load parameter of the computing device associated with the memory is acquired after a preset period is reached, and the comparison result obtained by comparing the load parameter with the reference parameter can be used to represent the load condition of the computing device under the current load parameter, so that the sliding window is adjusted according to the load condition reflected by the comparison result, and the data processing is performed using the adjusted sliding window, so that the load parameter of the computing device is consistent with the reference parameter of the computing device, and the computing device is adjusted to a load-balanced state.

[0054] According to an embodiment of the present application, the sliding window is dynamically adjusted based on the comparison result of the computing device load parameter and the reference parameter, so that the data processing process can adapt to the device load changes. This dynamic adjustment mechanism avoids performance degradation of the device due to data processing tasks when the load is too high, and fully utilizes the computing resources when the device is idle, achieving a balance between data processing efficiency and device load.

[0055] According to an embodiment of the present application, the data processing method further comprises: determining that the comparison result indicates that the computing device is in a load idle state if the number of times that the load parameter is lower than the reference parameter consecutively reaches a first number of times; and determining that the comparison result indicates that the computing device is in a load busy state if the number of times that the load parameter is higher than the reference parameter consecutively reaches a second number of times.

[0056] According to an embodiment of the present application, when the load parameter is lower than the reference parameter, it can be determined that the current computing device has a low load and more idle resources, and the resource input for data processing can be appropriately increased to improve processing efficiency and resource utilization.

[0057] According to an embodiment of the present application, when the load parameter is higher than the reference parameter, it can be determined that the current computing device has a high load and is in a load busy state, and the resource input for data processing can be appropriately reduced to maintain stable operation of the computing device.

[0058] According to an embodiment of the present application, since the load parameter of the computing device can fluctuate within a certain range, if the load parameter in the fluctuation is detected to be lower or higher than the reference parameter, the effect of directly adjusting the resource input for data processing will be poor, which is not conducive to load balancing of the computing device and efficiency of data processing.

[0059] Therefore, the first number of times and the second number of times can be set respectively to control the determination of the state of the computing device. Specifically, when the load parameter is lower than the reference parameter for the first number of times, it is determined that the comparison result indicates that the computing device is in a load idle state, and the resource input for data processing is further increased. Similarly, when the load parameter is higher than the reference parameter for the second number of times, it is determined that the comparison result indicates that the computing device is in a load busy state, and the resource input for data processing is appropriately reduced. The first number of times and the second number of times can be the same.

[0060] According to an embodiment of the present application, the load idle or busy state of the computing device is determined by consecutive number of times, which reduces the interference of single-time load fluctuation on state determination. This state determination method based on multiple detection results more accurately reflects the actual load of the device, provides a reliable basis for subsequent adjustment of the sliding window, and improves the accuracy of load adaptation.

[0061] According to an embodiment of the present application, the adjusting the sliding windows according to the comparison result comprises: in a case where the comparison result indicates that the computing device is in the load idle state, increasing the number of the sliding windows to process the unprocessed data blocks in parallel; and in a case where the comparison result indicates that the computing device is in the load busy state, reducing the number of the sliding windows and / or reducing the width of the sliding windows.

[0062] According to an embodiment of the present application, in a case where the comparison result indicates that the computing device is in the load idle state, the resource input for data processing can be appropriately increased. For example, the efficiency of data processing can be improved by increasing the parallelism of data processing. Specifically, the number of sliding windows can be increased, and the newly added sliding windows and the original sliding windows can be used to process the unprocessed data blocks in parallel.

[0063] According to an embodiment of the present application, in a case where the comparison result indicates that the computing device is in the load busy state, the resource input for data processing can be appropriately reduced. For example, the load of the computing device can be reduced by reducing the parallelism of data processing. Specifically, the number of sliding windows can be reduced, and the remaining sliding windows can be used to process the unprocessed data blocks.

[0064] For another example, in a case where the number of the current sliding windows cannot be reduced, the width of the sliding windows can be reduced to reduce the amount of data in the target data set to be processed each time, so as to release the computing resources and processing resources.

[0065] According to an embodiment of the present application, the number of sliding windows is increased to realize parallel processing when the device is in the load idle state, and the number of windows is reduced or the width of the windows is reduced when the device is in the load busy state, so as to adjust the parallelism and scale of data processing in a targeted manner. Such a differentiated adjustment strategy can not only accelerate the processing process when the computing device is idle, but also reduce the impact of processing tasks on the performance of the computing device when the computing device is busy, thereby optimizing the overall processing efficiency.

[0066] According to an embodiment of the present application, in a case where the comparison result indicates that the computing device is in the load idle state, the number of the sliding windows is increased to process the unprocessed data blocks in parallel, comprising: increasing the number of the sliding windows to M, M being an integer greater than 1; dividing the unprocessed data blocks to obtain M target data sets; and processing the M target data sets in parallel by using the M sliding windows respectively.

[0067] According to the embodiment of the present application, when the number of sliding windows is increased, the number of sliding windows can be increased from a current number to M, where the current number is less than M. The unprocessed data blocks to be processed are evenly divided into M target data sets of the same size, and the M sliding windows are used to process the M target data sets respectively, thereby improving the parallelism of the data blocks to be processed.

[0068] According to the embodiment of the present application, the number of sliding windows is increased to M, and the M target data sets are processed in parallel, thereby fully utilizing the computing power of the device when it is idle. The parallel processing mechanism significantly improves the data processing speed and shortens the overall processing time, and is particularly suitable for large-scale data processing scenarios, thereby improving the efficiency of data processing.

[0069] Figure 4 A schematic diagram of data processing by the data processing method according to the embodiment of the present application is shown.

[0070] As shown in Figure 4 , when the comparison result indicates that the computing device is in a load idle state, it is determined that the number of currently existing sliding windows is 1, and the sliding window is the first sliding window 431. The entire data block to be processed 400 is analyzed to determine the processed data block to be processed 410 and the unprocessed data block to be processed 420.

[0071] The number of sliding windows is increased by 1, a second sliding window 432 is set, and the unprocessed data block to be processed 420 is divided into two equal parts to obtain a first target data set 421 composed of unprocessed data blocks to be processed and a second target data set 422 composed of unprocessed data blocks to be processed. The first sliding window 431 is used to process the first target data set 421, and the second sliding window 432 is used to process the second target data set 422, thereby improving the parallelism of data processing and improving the efficiency of data processing.

[0072] According to the embodiment of the present application, when the comparison result indicates that the computing device is in a load busy state, the number of sliding windows is reduced and / or the width of the sliding window is reduced, including: when the number of sliding windows is greater than a preset number, the number of sliding windows is reduced to N, where N is an integer greater than or equal to 1; dividing the unprocessed data block to be processed to obtain N target data sets; using N sliding windows to process N target data sets in parallel; when the number of sliding windows is not greater than the preset number, reducing the width of the sliding window based on the width of the sliding window and a preset ratio.

[0073] According to the embodiment of the present application, the preset number can be set according to the minimum parallelism of data processing that the user can accept, and the default value of the preset number can be 1 when the user does not set the preset number.

[0074] According to an embodiment of the present application, in the case that the number of sliding windows is greater than the preset number, the number of sliding windows is reduced to N. Wherein, the current number of sliding windows is greater than N. The logic of sliding window number reduction can be set to reduce by a preset fixed value or by a preset reduction ratio. For example, the preset fixed value can be 1, and the number of sliding windows is reduced by 1 each time, and the preset reduction ratio can be 10%, and the number of sliding windows required to be reduced this time is calculated based on the current number of sliding windows each time, and in the case that the calculated required reduction number is not an integer, the calculation result is rounded up to make each adjustment at least reduce 1 sliding window.

[0075] According to an embodiment of the present application, in the case that the number of sliding windows is not greater than the preset number, for example, only 1 sliding window is left, at this time, if the number of sliding windows continues to be reduced, the data processing will be interrupted, affecting the normal data processing process and efficiency. The width of the sliding window can be reduced according to the preset ratio and the width of the sliding window. For example, the preset ratio can be 20%, and after the width of the sliding window with a width of W is reduced, the adjusted sliding window has a width of 0.8W.

[0076] According to an embodiment of the present application, in the case that the number of sliding windows is not greater than the preset number, for example, only 1 sliding window is left, at this time, if the number of sliding windows continues to be reduced, the data processing will be interrupted, affecting the normal data processing process and efficiency. The width of the sliding window can be reduced according to the preset ratio and the width of the sliding window. For example, the preset ratio can be 20%, and after the width of the sliding window with a width of W is reduced, the adjusted sliding window has a width of 0.8W.

[0077] According to another embodiment of the present application, in the moving process of the sliding window, although a plurality of unprocessed data blocks to be processed can be logically placed in the same target data set, the physical positions of the data blocks to be processed in the memory can actually not be adjacent. After the current target data set is determined by the sliding window, the storage addresses of the unprocessed data blocks to be processed in the target data set in the memory can be counted, and the subsequent moving mode of the sliding window can be planned according to the storage addresses.

[0078] For example, in the case where the subsequent unprocessed to-be-processed data blocks are adjacent, the sliding window can be moved by a preset step size, so as to determine a new target data set according to the to-be-processed data blocks covered by the moved sliding window. In the case where the subsequent unprocessed to-be-processed data blocks are not adjacent, a shortest path algorithm can be applied on the memory, and the to-be-processed data blocks are sequentially selected in the order of the path from near to far until the width of the sliding window is satisfied, and a new target data set is determined. Therefore, the sliding path of the sliding window can be optimized without changing the block logic of the to-be-processed data, the width of the sliding window, and the sliding logic, so as to reduce the processing delay caused by the scattered distribution of the to-be-processed data blocks, and improve the efficiency of data processing.

[0079] According to an embodiment of the present application, the data processing method further comprises: obtaining the number of target data sets currently existing in the computing device, the load parameter of the computing device, and the processing speed of the computing device for the to-be-processed data blocks; and recording the number of target data sets, the load parameter, and the processing speed respectively according to a first period.

[0080] According to an embodiment of the present application, since each sliding window is used to process a target data set during data processing, the number of target data sets currently existing in the computing device can be used to determine the number of sliding windows currently existing, and the parallelism of the current data processing can be determined.

[0081] According to an embodiment of the present application, the data processing method further comprises: determining the processing time consumption according to the difference between the division time when the to-be-processed data is divided into a plurality of to-be-processed data blocks and the current time; and determining the processing speed according to the ratio of the number of to-be-processed data blocks that have been processed to the processing time consumption.

[0082] According to an embodiment of the present application, after the to-be-processed data is divided into a plurality of to-be-processed data blocks, the processing phase of the to-be-processed data blocks is entered, and therefore, the processing time consumption consumed for processing the to-be-processed data blocks can be determined according to the difference between the current time and the division time.

[0083] According to an embodiment of the present application, the processing speed of the to-be-processed data blocks can be determined according to the processing time consumption and the number of to-be-processed data blocks that have been processed.

[0084] According to an embodiment of the present application, the processing speed is determined based on the difference between the division completion time and the current time and the number of processed data blocks, and the efficiency index of data processing is quantified. Accurate calculation of the processing speed provides an objective basis for evaluating the processing progress and adjusting the processing strategy, and helps to more scientifically plan the data processing process.

[0085] The number of target datasets, the load parameters of computing devices, and the current processing speed of the data blocks to be processed can be stored in a data table or log file for recording data processing, so that the data processing history can be queried when needed.

[0086] According to embodiments of this application, the number of target datasets, load parameters, and processing speed are recorded in a first cycle, forming real-time monitoring and data accumulation of the data processing process. These recorded parameters provide data support for analyzing processing efficiency and optimizing processing strategies, facilitate tracing problems during processing, and improve the manageability of data processing.

[0087] According to an embodiment of this application, the data processing method further includes: updating the baseline parameters according to a second cycle, using the number of target datasets, load parameters, and processing speed, wherein the second cycle is longer than the first cycle.

[0088] According to embodiments of this application, since the performance conditions of a computing device may change due to hardware aging, driver upgrades, hardware replacements, etc., the baseline parameters of the computing device under load balancing will also change. The baseline parameters of the computing device under load balancing can be periodically recalculated and updated based on the processing speed performance of the computing device under different load parameters and different data processing parallelism.

[0089] According to an embodiment of this application, the second cycle is longer than the first cycle, so as to ensure that the number of recorded target datasets, load parameters and processing speed have been updated each time the baseline parameters are updated, and to avoid updating the baseline parameters before the above parameters have been updated, thus wasting computing resources due to invalid updates.

[0090] According to embodiments of this application, the baseline parameters are updated using recorded parameters in a second cycle, enabling the baseline parameters to dynamically adapt to changes in device state during data processing. The updated baseline parameters better reflect the actual operating conditions of the device, improving the accuracy of load assessment and window adjustment, and enhancing the adaptability of the method.

[0091] According to an embodiment of this application, the data processing method further includes: overwriting the data in the target dataset multiple times to clear the original data in the target dataset, including: overwriting the original data in the target dataset using a first overwrite value, such that all data in the target dataset is the first overwrite value; overwriting the data in the target dataset using a second overwrite value, such that all data in the target dataset is the second overwrite value; and overwriting the data in the target dataset using a random value to clear the original data in the target dataset.

[0092] According to an embodiment of the present application, the first cover value can be 0, and the second cover value can be 1. By using the first cover value to overwrite, the problem of incomplete overwriting caused by data residues can be avoided.

[0093] Taking a mechanical hard disk as an example, by using the first cover value to overwrite for the first time, all storage locations can be magnetized to a "0" state, and the magnetic field strength is uniformly lowered by writing 0 to eliminate the residual traces of "1" in the original data. Similarly, by using the second cover value to overwrite for the second time, all storage locations can be magnetized to a "1" state, and the magnetic field strength is uniformly raised by writing 1 to eliminate the residual traces of "0" in the original data, and to ensure that the physical state of all locations tends to be consistent, reducing the possibility of subsequent recovery by technical means, thereby ensuring that the data to be processed is completely cleared, improving data security and the credibility of data processing.

[0094] According to an embodiment of the present application, after the above-mentioned two cover writings are completed, a random value can be used to overwrite the data in the target data set to cover with irregular 0s and 1s, breaking any possible pattern residues in the memory and ensuring that the original data is difficult to be interpreted and recovered from the residual traces.

[0095] According to another embodiment of the present application, a preset number of overwrites can be set, and random value overwrites of the preset number of overwrites can be performed, thereby further reducing the regularity of the data in the target data set and improving the effect of data clearing.

[0096] According to an embodiment of the present application, the target data set is overwritten multiple times by the first cover value, the second cover value and the random value, and multiple rounds and multiple types of overwriting operations ensure that the original data is completely cleared. This multi-level clearing mechanism effectively prevents data residues and recovery risks, and improves the security and completeness of data clearing.

[0097] According to an embodiment of the present application, the data processing method further comprises: in response to completion of dividing the data to be processed into a plurality of data blocks arranged in sequence, setting the use state of the storage subspaces corresponding to the plurality of data blocks to be processed to be unavailable; and in response to clearing the original data of the target data set, setting the use state of the storage subspaces of at least one data block to be processed in the target data set to be available.

[0098] According to an embodiment of the present application, the use state of the storage subspaces is used to indicate whether the corresponding storage subspaces can be currently used for data storage or data read / write. After the data to be processed is divided into a plurality of data blocks to be processed, since the plurality of data blocks to be processed are data blocks that need to be cleared and have not yet completed data clearing, the use state of the storage subspaces corresponding to the plurality of data blocks to be processed is set to be unavailable before the data clearing is completed.

[0099] According to the embodiments of the present application, after the original data of the target data set is cleared, the use state of the storage sub-space of at least one to-be-processed data block in the target data set is set to available, so as to release the storage sub-space in time.

[0100] According to the embodiments of the present application, the storage sub-space is set to unavailable after the data block division is completed, and is restored to available after the data is cleared. The state management ensures the reasonable use of the storage resource in the data processing process. The misoperation on the storage sub-space before the data is completely cleared is prevented, and the resource is released in time after the processing is completed, so that the utilization rate of the storage resource is improved.

[0101] In the case that the user has a high degree of understanding of the performance of the computing device associated with the storage or has a preliminary requirement on the efficiency of the data processing, the user can set the initial parallelism of the data processing. Generally, the initial parallelism set by the user is closer to the performance of the computing device than the default parallelism, and is more consistent with the benchmark parameter.

[0102] Although according to the above adjustment method of the sliding window, the data processing of any parallelism can be adjusted to the case that the computing device is balanced, if the initial parallelism is greatly different from the parallelism that balances the computing device, there will be problems such as low data processing efficiency or unstable operation of the computing device in the adjustment process.

[0103] Therefore, using the initial parallelism set by the user can reduce the problems such as low data processing efficiency or unstable operation of the computing device in the process of aligning the load parameter corresponding to the default parallelism to the benchmark parameter, further improve the data processing efficiency, maintain the stable operation of the computing device, and improve the user experience.

[0104] Specifically, the data processing method further comprises: in response to the custom instruction, dividing the to-be-processed data block into P target data sets, P is determined according to the custom instruction, and P is an integer greater than or equal to 1; and processing the P target data sets in parallel by using P sliding windows.

[0105] According to the embodiments of the present application, the user can issue a custom instruction according to the use experience or processing requirement, so as to determine the parallelism P according to the custom instruction, divide the to-be-processed data block into P equal parts to obtain P target data sets, and process the P target data sets in parallel by using P sliding windows.

[0106] According to an embodiment of the present application, the data block is divided into P target data sets in response to the custom instruction, and the P target data sets are processed in parallel, supporting flexible adjustment of the parallel degree of processing according to user needs. This custom mechanism enhances the flexibility and applicability of the method, meets the data processing needs in different scenarios, and improves the user experience.

[0107] Figure 5 A flowchart of a data processing method according to another embodiment of the present application is shown.

[0108] As shown in Figure 5 , the method includes operations S501-S517.

[0109] In operation S501, it is determined whether there is a custom instruction of a user. If yes, operation S502 is performed; if no, operation S503 is performed.

[0110] In operation S502, a custom parallel degree is set according to the custom instruction.

[0111] In operation S503, a default parallel degree is set.

[0112] In operation S504, a data processing instruction is received.

[0113] In operation S505, based on the data processing instruction, a storage space in which the to-be-processed data is stored is determined from a memory.

[0114] In operation S506, the to-be-processed data is divided into a plurality of to-be-processed data blocks according to the storage space in which the to-be-processed data is located.

[0115] In operation S507, the number of sliding windows and the number of target data sets are determined according to the current parallel degree.

[0116] In operation S508, the target data sets are processed using the sliding windows.

[0117] In operation S509, it is determined whether a preset period is reached. If yes, operation S510 is performed; if no, operation S509 is returned.

[0118] In operation S510, a load parameter of a computing device is obtained, and the load parameter is compared with a reference parameter.

[0119] In operation S511, it is determined whether the comparison result indicates that the current computing device is in a load balancing state. If yes, operation S509 is returned; if no, operation S512 is performed.

[0120] In operation S512, determine whether the computing device is in an idle state. If it is in an idle state, execute operation S513; if it is not in an idle state, it is in a busy state, and execute operation S514.

[0121] In operation S513, the number of sliding windows is increased to improve parallelism. Then, operation S517 is executed.

[0122] In operation S514, determine whether the number of sliding windows is greater than a preset number. If it is greater than the preset number, execute operation S515; otherwise, execute operation S516.

[0123] In operation S515, reduce the number of sliding windows. Execute operation S517.

[0124] In operation S516, reduce the width of the sliding window. Then execute operation S517.

[0125] When operating S517, the target dataset is processed using the adjusted sliding window.

[0126] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0127] Figure 6 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0128] like Figure 6 As shown, the data processing device 600 of this embodiment includes a space determination module 610, a data partitioning module 620, and a data block processing module 630. The data block processing module 630 includes a data block selection submodule 631, a data overwrite submodule 632, and a window sliding submodule 633.

[0129] The space determination module 610 is used to determine, in response to receiving a data processing instruction, the storage space in the memory where the data to be processed is stored, based on the data processing instruction. In one embodiment, the space determination module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0130] The data partitioning module 620 is used to divide the data to be processed into multiple sequentially arranged data blocks according to their storage addresses in the storage space. The data blocks have the same amount of data. In one embodiment, the data partitioning module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0131] The data block processing module 630 is configured to process the plurality of to-be-processed data blocks in sequence by using a sliding window. In an embodiment, the data block processing module 630 can be configured to perform the operation S230 described above, and details are not repeated here.

[0132] The data block selection submodule 631 is configured to select at least one to-be-processed data block as a target data set by using a sliding window, where the width of the sliding window is an integer multiple of the data amount of the to-be-processed data block. In an embodiment, the data block selection submodule 631 can be configured to perform the operation S231 described above, and details are not repeated here.

[0133] The data overwrite submodule 632 is configured to overwrite the data in the target data set for multiple times to clear the original data of the target data set. In an embodiment, the data overwrite submodule 632 can be configured to perform the operation S232 described above, and details are not repeated here.

[0134] The window sliding submodule 633 is configured to move the sliding window by a preset step to perform the next processing in the case that there is an unprocessed to-be-processed data block, where the preset step is related to the width of the sliding window. In an embodiment, the window sliding submodule 633 can be configured to perform the operation S233 described above, and details are not repeated here.

[0135] According to an embodiment of the present application, the data processing apparatus 600 further comprises a parameter comparison module and a window adjustment module.

[0136] The parameter comparison module is configured to periodically acquire a load parameter of a computing device associated with the memory, and compare the load parameter of the computing device with a reference parameter of the computing device to obtain a comparison result.

[0137] The window adjustment module is configured to adjust the sliding window according to the comparison result, so that the load parameter of the computing device is consistent with the reference parameter of the computing device in the case that the processing is performed by using the adjusted sliding window.

[0138] According to an embodiment of the present application, the data processing apparatus 600 further comprises a first state determination module and a second state determination module.

[0139] The first state determination module is configured to determine that the comparison result indicates that the computing device is in a load idle state in the case that the number of times that the load parameter is lower than the reference parameter continuously appears reaches a first number of times.

[0140] The second state determination module is configured to determine that the comparison result indicates that the computing device is in a load busy state in the case that the number of times that the load parameter is higher than the reference parameter continuously appears reaches a second number of times.

[0141] According to an embodiment of the present application, the window adjustment module comprises a first window adjustment submodule and a second window adjustment submodule.

[0142] The first window adjustment submodule is configured to increase the number of sliding windows to process the unprocessed data blocks in parallel when the comparison result indicates that the computing device is in the load idle state.

[0143] The second window adjustment submodule is configured to reduce the number of sliding windows and / or reduce the width of the sliding windows when the comparison result indicates that the computing device is in the load busy state.

[0144] According to an embodiment of the present application, the first window adjustment submodule comprises a window increasing unit, a first data dividing unit and a first data processing unit.

[0145] The window increasing unit is configured to increase the number of sliding windows to M, M being an integer greater than 1.

[0146] The first data dividing unit is configured to divide the unprocessed data blocks to obtain M target data sets.

[0147] The first data processing unit is configured to process the M target data sets in parallel using the M sliding windows respectively.

[0148] According to an embodiment of the present application, the second window adjustment submodule comprises a window reducing unit, a second data dividing unit, a second data processing unit and a window width reducing unit.

[0149] The window reducing unit is configured to reduce the number of sliding windows to N when the number of sliding windows is greater than a preset number, N being an integer greater than or equal to 1.

[0150] The second data dividing unit is configured to divide the unprocessed data blocks to obtain N target data sets.

[0151] The second data processing unit is configured to process the N target data sets in parallel using the N sliding windows respectively.

[0152] The window width reducing unit is configured to reduce the width of the sliding windows based on the width of the sliding windows and a preset ratio when the number of sliding windows is not greater than the preset number.

[0153] According to an embodiment of the present application, the data processing apparatus 600 further comprises a parameter obtaining module and a parameter recording module.

[0154] The parameter obtaining module is configured to obtain the number of target data sets currently existing in the computing device, the load parameter of the computing device and the processing speed of the computing device for the data blocks.

[0155] The parameter recording module is configured to record the number of target data sets, the load parameter, and the processing speed respectively according to a first period.

[0156] According to an embodiment of the present application, the data processing apparatus 600 further comprises a time consumption determining module and a speed determining module.

[0157] The time consumption determining module is configured to determine the processing time consumption according to a difference between a division time and a current time, the division time being when the to-be-processed data is divided into a plurality of to-be-processed data blocks.

[0158] The speed determining module is configured to determine the processing speed according to a ratio of the number of to-be-processed data blocks that have been processed to the processing time consumption.

[0159] According to an embodiment of the present application, the data processing apparatus 600 further comprises a parameter updating module.

[0160] The parameter updating module is configured to update the reference parameter by using the number of target data sets, the load parameter, and the processing speed according to a second period, wherein the second period is greater than the first period.

[0161] According to an embodiment of the present application, the data overwriting submodule 632 comprises a first overwriting unit, a second overwriting unit, and a third overwriting unit.

[0162] The first overwriting unit is configured to overwrite the original data in the target data set by using the first cover value, so that the data in the target data set are all the first cover value.

[0163] The second overwriting unit is configured to overwrite the data in the target data set by using the second cover value, so that the data in the target data set are all the second cover value.

[0164] The third overwriting unit is configured to overwrite the data in the target data set by using a random value, so as to clear the original data of the target data set.

[0165] According to an embodiment of the present application, the data processing apparatus 600 further comprises a first state setting module and a second state setting module.

[0166] The first state setting module is configured to set usage states of storage subspaces corresponding to the plurality of to-be-processed data blocks as unavailable in response to completion of dividing the to-be-processed data into the plurality of to-be-processed data blocks arranged in sequence.

[0167] The second state setting module is configured to set a usage state of a storage subspace of at least one to-be-processed data block in the target data set as available in response to clearing the original data of the target data set.

[0168] According to an embodiment of the present application, the data processing apparatus 600 further comprises an instruction response module and a parallel processing module.

[0169] The instruction response module is configured to divide the to-be-processed data block into P target data sets in response to the customized instruction, where P is determined according to the customized instruction, and P is an integer greater than or equal to 1.

[0170] The parallel processing module is configured to perform parallel processing on the P target data sets by using the P sliding windows respectively.

[0171] According to an embodiment of the present application, the space determination module 610, the data division module 620 and the data block processing module 630, wherein the data block processing module 630 includes any number of modules among the data block selection submodule 631, the data overwrite submodule 632 and the window sliding submodule 633 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present application, the space determination module 610, the data division module 620 and the data block processing module 630, wherein at least one of the data block processing module 630 includes the data block selection submodule 631, the data overwrite submodule 632 and the window sliding submodule 633 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system in package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging circuits, etc. hardware or firmware, or any one of the three implementation ways of software, hardware and firmware or any appropriate combination of several of them. Alternatively, the space determination module 610, the data division module 620 and the data block processing module 630, wherein at least one of the data block processing module 630 includes the data block selection submodule 631, the data overwrite submodule 632 and the window sliding submodule 633 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0172] Figure 7 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present application is shown.

[0173] As Figure 7As shown, the electronic device 700 according to embodiments of the present application includes a processor 701 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 702 or a program loaded into a random access memory (RAM) 703 from a storage section 708. The processor 701 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0174] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 702 and / or the RAM 703. Note that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0175] According to embodiments of the present application, the electronic device 700 can also include an input / output (I / O) interface 705 which is also connected to the bus 704. The electronic device 700 can also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as necessary. A removable medium 711 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read out therefrom is installed in the storage section 708 as necessary.

[0176] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0177] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include the ROM 702 and / or the RAM 703 described above, and / or one or more memory other than the ROM 702 and the RAM 703.

[0178] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the application.

[0179] The above functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 701. According to the embodiments of the application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0180] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 709, and / or installed from the detachable medium 711. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0181] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 709, and / or installed from the removable media 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be realized by the computer program module.

[0182] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0183] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the figure. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0184] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations fall within the scope of the present application.

[0185] The embodiments of the application have been described. However, these embodiments are merely for illustration and are not intended to limit the scope of the application. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Various alternatives and modifications to the embodiments described herein will be apparent to those skilled in the art in view of the foregoing description. Such alternatives and modifications are intended to fall within the scope of the application.

Claims

1. A data processing method, characterized in that, The method includes: In response to receiving a data processing instruction, the storage space in the memory containing the data to be processed is determined based on the data processing instruction; According to the storage address in the storage space, the data to be processed is divided into multiple data blocks arranged in sequence, and the data volume of the multiple data blocks to be processed is the same. Using a sliding window, the plurality of data blocks to be processed are processed sequentially, wherein any single processing step includes: Using the sliding window, at least one of the data blocks to be processed is selected as the target dataset, wherein the width of the sliding window is an integer multiple of the data size of the data block to be processed; The data in the target dataset is overwritten multiple times to clear the original data in the target dataset; In the presence of unprocessed data blocks, the sliding window is moved by a preset step size to perform the next processing step, wherein the preset step size is related to the width of the sliding window; The step of repeatedly overwriting the data in the target dataset to clear the original data in the target dataset includes: Using the first overwrite value, the original data in the target dataset is overwritten so that all the data in the target dataset is the first overwrite value; Using the second overwrite value, the data in the target dataset is overwritten so that all data in the target dataset contains the second overwrite value; and The data in the target dataset is overwritten using random values ​​to clear the original data in the target dataset; The method further includes: Periodically acquire the load parameters of the computing device associated with the memory, and compare the load parameters of the computing device with the baseline parameters of the computing device to obtain the comparison result; and Based on the comparison results, the sliding window is adjusted so that when processing is performed using the adjusted sliding window, the load parameters of the computing device are consistent with the baseline parameters of the computing device. Based on the comparison results, adjust the sliding window, including: If the comparison result indicates that the computing device is in an idle state, the number of sliding windows is increased to process unprocessed data blocks in parallel, wherein the computing device is determined to be in an idle state when the load parameter is lower than the baseline parameter for the first time; and If the comparison result indicates that the computing device is under heavy load, reduce the number of sliding windows and / or reduce the width of the sliding windows, wherein the computing device is determined to be under heavy load if the load parameter is higher than the baseline parameter by a second number.

2. The method according to claim 1, characterized in that, The method further includes: If the number of times the load parameter is lower than the reference parameter reaches the first count, it is determined that the comparison result indicates that the computing device is in an idle state; and If the number of times the load parameter is higher than the baseline parameter reaches a second consecutive count, the comparison result indicates that the computing device is in a state of heavy load.

3. The method according to claim 1, characterized in that, When the comparison result indicates that the computing device is in an idle state, the number of sliding windows is increased to process unprocessed data blocks in parallel, including: Increase the number of sliding windows to M, where M is an integer greater than 1; The unprocessed data block is divided into M target datasets; and The M target datasets are processed in parallel using M sliding windows respectively.

4. The method according to claim 1, characterized in that, When the comparison result indicates that the computing device is under heavy load, reducing the number of sliding windows and / or reducing the width of the sliding windows includes: If the number of sliding windows is greater than a preset number, the number of sliding windows is reduced to N, where N is an integer greater than or equal to 1. The unprocessed data block is divided into N target datasets; The N target datasets are processed in parallel using N sliding windows respectively; If the number of sliding windows is not greater than the preset number, the width of the sliding window is reduced based on the width of the sliding window and the preset ratio.

5. The method according to claim 3 or 4, characterized in that, The method further includes: The system obtains the number of target datasets currently existing in the computing device, the load parameters of the computing device, and the current processing speed of the computing device for the data block to be processed; and Record the number of target datasets, the load parameters, and the processing speed in the first cycle.

6. The method according to claim 5, characterized in that, The method further includes: The processing time is determined by the difference between the time when the data to be processed was divided into multiple data blocks and the current time. The processing speed is determined based on the ratio of the number of processed data blocks to the processing time.

7. The method according to claim 5, characterized in that, The method further includes: The baseline parameters are updated according to the second cycle, using the number of the target dataset, the load parameters, and the processing speed, wherein the second cycle is longer than the first cycle.

8. The method according to claim 1, characterized in that, The method further includes: In response to the completion of dividing the data to be processed into multiple sequentially arranged data blocks, the usage status of the storage subspace corresponding to each of the multiple data blocks to be processed is set to unavailable; In response to clearing the original data in the target dataset, the usage status of the storage subspace of at least one of the data blocks to be processed in the target dataset is set to available.

9. The method according to claim 1, characterized in that, The method further includes: In response to a custom instruction, the data block to be processed is divided into P target datasets, where P is determined according to the custom instruction and is an integer greater than or equal to 1. P target datasets are processed in parallel using P sliding windows.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images