Screening method and device of disk data, computer equipment and storage medium

Abstract

The invention relates to a disk data screening method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that disk attribute information of a to-be-tested disk is obtained, the disk attribute information comprises disk state information of the to-be-tested disk and data attribute information of multiple pieces of disk data stored in the to-be-tested disk, data integrity analysis is carried out on the disk data according to the disk state information and the data attribute information, and the disk data is obtained; according to the method, multiple pieces of candidate data with the expected integrity smaller than the integrity threshold value are screened from the disk data, primary screening is conducted on the disk data, cyclic redundancy check is conducted on the candidate data for each piece of candidate data, and the check result of the candidate data is obtained. And the target data which does not pass the check result is determined from the candidate data, so that the data volume of cyclic redundancy check is reduced, and the check efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of disk inspection technology, and in particular to a disk data screening method, device, computer equipment, storage medium and computer program product. Background Art

[0002] Disks are widely used in scenarios involving data storage. The data stored in disks is vital to enterprises. Once the data in disks is damaged or lost, it may lead to serious consequences such as business interruption, information leakage or economic loss. In order to ensure the accuracy and reliability of stored data, data integrity verification is required for the stored data in disks.

[0003] In the conventional technology, the integrity of each piece of stored data in the disk is verified in sequence to obtain the integrity of each piece of stored data, which has the problem of low verification efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a disk data screening method, device, computer equipment, computer-readable storage medium and computer program product that can improve verification efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for screening disk data. The method comprises:

[0006] Obtain disk attribute information of the disk to be tested, the disk attribute information including disk status information of the disk to be tested and data attribute information of each of multiple disk data stored on the disk to be tested; perform data integrity analysis on each of the disk data according to the disk status information and each of the data attribute information to obtain the expected integrity of each of the disk data; screen multiple candidate data whose expected integrity is less than an integrity threshold among each of the disk data; perform cyclic redundancy check on each of the candidate data to obtain the verification result of the candidate data; and determine the target data whose verification result fails from each of the candidate data.

[0007] In one of the embodiments, data integrity analysis is performed on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data, including: extracting features from the disk status information to obtain disk status features of the disk to be tested; extracting features from the data attribute information to obtain data attribute features of the disk data; inputting the disk status features and the data attribute features into an integrity analysis model, and determining the predicted integrity of the disk data based on the output of the integrity analysis model.

[0008] In one embodiment, the training process of the completeness analysis model includes: obtaining an initial model and multiple disk data samples carrying completeness labels; performing model training on the initial model based on each of the disk data samples to obtain a predicted completeness of each of the disk data samples; determining an actual completeness ranking and a predicted completeness ranking of each of the disk data samples based on the completeness label and predicted completeness of each of the disk data samples; determining the difference between the actual completeness ranking and the predicted completeness ranking, and returning to the step of performing model training on the initial model based on each of the disk data samples to perform the next round of iteration; and obtaining a completeness analysis model when the training end conditions are met.

[0009] In one of the embodiments, when the training end condition is met, a completeness analysis model is obtained, including: obtaining the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round; when the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the historical difference have a coefficient of variation within a preset round that is less than a preset value, a completeness analysis model is obtained.

[0010] In one of the embodiments, the method further includes: obtaining a computing capability corresponding to the disk to be tested; determining a completeness threshold that matches the computing capability; and the completeness threshold is positively correlated with the computing capability.

[0011] In one of the embodiments, obtaining the computing capacity corresponding to the disk to be tested includes: determining a terminal used to process the disk to be tested; obtaining the disk computing capacity of the disk to be tested and the terminal computing capacity of the terminal; and determining the computing capacity corresponding to the disk to be tested based on the disk computing capacity and the terminal computing capacity.

[0012] In a second aspect, the present application also provides a device for screening disk data. The device comprises:

[0013] An acquisition module is used to acquire disk attribute information of the disk to be tested, wherein the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of the multiple disk data stored on the disk to be tested; an analysis module is used to perform data integrity analysis on each of the disk data according to the disk status information and each of the data attribute information to obtain the expected integrity of each of the disk data; a screening module is used to screen multiple candidate data whose expected integrity is less than a integrity threshold from each of the disk data; a verification module is used to perform a cyclic redundancy check on each of the candidate data to obtain the verification result of the candidate data; a processing module is used to determine the target data whose verification result fails from each of the candidate data.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0015] Obtain disk attribute information of the disk to be tested, the disk attribute information including disk status information of the disk to be tested and data attribute information of each of multiple disk data stored on the disk to be tested; perform data integrity analysis on each of the disk data according to the disk status information and each of the data attribute information to obtain the expected integrity of each of the disk data; screen multiple candidate data whose expected integrity is less than an integrity threshold among each of the disk data; perform cyclic redundancy check on each of the candidate data to obtain the verification result of the candidate data; and determine the target data whose verification result fails from each of the candidate data.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0017] Obtain disk attribute information of the disk to be tested, the disk attribute information including disk status information of the disk to be tested and data attribute information of each of multiple disk data stored on the disk to be tested; perform data integrity analysis on each of the disk data according to the disk status information and each of the data attribute information to obtain the expected integrity of each of the disk data; screen multiple candidate data whose expected integrity is less than an integrity threshold among each of the disk data; perform cyclic redundancy check on each of the candidate data to obtain the verification result of the candidate data; and determine the target data whose verification result fails from each of the candidate data.

[0018] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0019] Obtain disk attribute information of the disk to be tested, the disk attribute information including disk status information of the disk to be tested and data attribute information of each of multiple disk data stored on the disk to be tested; perform data integrity analysis on each of the disk data according to the disk status information and each of the data attribute information to obtain the expected integrity of each of the disk data; screen multiple candidate data whose expected integrity is less than an integrity threshold among each of the disk data; perform cyclic redundancy check on each of the candidate data to obtain the verification result of the candidate data; and determine the target data whose verification result fails from each of the candidate data.

[0020] The above-mentioned method, device, computer equipment, storage medium and computer program product for screening disk data obtain the disk attribute information of the disk to be tested, the disk attribute information includes the disk status information of the disk to be tested, and the data attribute information of each of the multiple disk data stored on the disk to be tested, and the data integrity analysis is performed on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data, which reflects the estimated integrity of each disk data, which is conducive to the subsequent screening of each data and improves the verification efficiency of the disk data. Using the above-mentioned method, in each disk data, multiple candidate data with expected integrity less than the integrity threshold are screened, each disk data is first preliminarily screened, and then for each candidate data, a cyclic redundancy check is performed on the candidate data to obtain the verification result of the candidate data, and the target data whose verification result fails is determined from each candidate data, thereby reducing the amount of data for the cyclic redundancy check and improving the verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is an application environment diagram of a method for screening disk data in one embodiment;

[0022] FIG2 is a schematic diagram of a flow chart of a method for screening disk data in one embodiment;

[0023] FIG3 is a flow chart of a method for training a completeness analysis model in one embodiment;

[0024] FIG4 is a schematic diagram of the structure of an initial model in one embodiment;

[0025] FIG5 is a schematic diagram of the structure of a state space in one embodiment;

[0026] FIG6 is a schematic diagram of a flow chart of a disk data review method based on artificial intelligence in one embodiment;

[0027] FIG7 is a flow chart of a disk data review method based on artificial intelligence in another embodiment;

[0028] FIG8 is a schematic diagram of the structure of a data collection and preprocessing module in one embodiment;

[0029] FIG9 is a block diagram of a device for screening disk data in one embodiment;

[0030] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The method for screening disk data provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . The disk to be tested 102 is in communication connection with the inspection terminal 104 . The data storage system can store the data to be processed by the inspection terminal 104 . The data storage system can be integrated on the inspection terminal 104 .

[0033] The inspection terminal 104 obtains disk attribute information of the disk to be tested 102, and the disk attribute information includes disk status information of the disk to be tested 102 and data attribute information of each of the multiple disk data stored in the disk to be tested 102. The inspection terminal 104 performs data integrity analysis on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data. The inspection terminal 104 selects multiple candidate data whose expected integrity is less than the integrity threshold from each disk data. The inspection terminal 104 performs a cyclic redundancy check on each candidate data to obtain the verification result of the candidate data. The inspection terminal 104 determines the target data whose verification result fails from each candidate data.

[0034] The disk 102 to be tested may be a memory that stores data using magnetic recording technology.

[0035] The inspection terminal 104 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Specifically, the inspection terminal 104 may communicate with the disk to be tested 102, and the inspection terminal 104 may also communicate with the terminal to which the disk to be tested 102 belongs.

[0036] In one embodiment, as shown in FIG. 2 , a method for screening disk data is provided, which is described by taking the method applied to the inspection terminal 104 in FIG. 1 as an example, including:

[0037] S202: Obtain disk attribute information of the disk to be tested.

[0038] The disk may be a memory that stores data using magnetic recording technology, and can store a large amount of binary data, and the data can be retained even after a power outage.

[0039] The disk to be tested may be a disk that needs to be tested for integrity. Specifically, in order to ensure the data security of the disk, prevent malicious tampering, detect and repair errors, and improve system reliability, the disk to be tested may be tested for integrity. For example, CRC check, hash check, and parity check in RAID technology, etc.

[0040] The disk attribute information may represent information on verification-related dimensions of the disk to be tested.

[0041] The disk attribute information may include disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested.

[0042] The disk status information indicates the status information of the disk to be tested, for example, the SMART status information of the disk to be tested and the data damage rate of the disk to be tested.

[0043] Among them, the SMART status information of the disk to be tested can represent the self-monitoring, analysis and reporting technology of the disk. The SMART status includes a series of parameters and information to reflect the health and working status of the hard disk. The main SMART parameters include at least: attribute ID, attribute name, attribute identification, manufacturer detection custom content, etc. The attribute ID is used to uniquely identify the SMART attribute. The attribute name can represent the name of a certain test item and is a textual explanation of the ID code. The attribute identification can be a mark carried by the attribute, such as offline collection, online collection, key data, normal wear attributes, performance-related attributes, etc. The manufacturer detection custom content can represent the current value, worst value, critical value, etc. of the disk detection items defined by the disk manufacturer.

[0044] The disk data may represent specific data stored in the disk to be tested. The disk to be tested includes multiple pieces of disk data. The data attribute information may represent attribute information of the disk data. For example, the data attribute information may include the data size, write time, previous read times, read times, previous review times, user level corresponding to the data user, etc. of the disk data.

[0045] Specifically, the inspection terminal may obtain disk attribute information of the disk to be tested.

[0046] In one embodiment, the test terminal can directly read the disk attribute information from the disk under test, or can also indirectly read the disk attribute information from the terminal under test to which the disk under test belongs.

[0047] In one embodiment, due to the computing power limitation of the inspection terminal, the inspection terminal may read the data attribute information of each of the partial disk data in the disk to be tested. Specifically, the inspection terminal may determine the read data volume of the disk data according to its own computing power. The inspection terminal reads the data attribute information of each of the plurality of disk data matching the read data volume in the disk to be tested.

[0048] S204, performing data integrity analysis on each disk data according to the disk status information and each data attribute information, and obtaining the expected integrity of each disk data.

[0049] The expected completeness may represent the predicted completeness of the disk data. It is understandable that if the expected completeness of the disk data is greater, the possibility of data missing in the disk data is lower. Conversely, if the expected completeness of the disk data is smaller, the possibility of data missing in the disk data is higher.

[0050] There are many ways to perform data integrity analysis on disk data. Specifically, the inspection terminal can obtain standard disk status information and standard data attribute information. The inspection terminal can compare the status information difference between the disk status information and the standard disk status information, and can compare the attribute information difference between each data attribute information and the standard data attribute information. The inspection terminal can determine the expected integrity of each disk data based on the status information difference and each attribute information difference. The greater the status information difference and the attribute information difference, the smaller the expected integrity of the corresponding disk data.

[0051] Specifically, the inspection terminal can also perform feature extraction on the disk status information to obtain the disk status feature of the disk to be tested. The inspection terminal can perform feature extraction on the data attribute information to obtain the data attribute feature of the disk data. The inspection terminal can input the disk status feature and the data attribute feature into the integrity analysis model, and determine the predicted integrity of the disk data based on the output of the integrity analysis model.

[0052] S206: Filter multiple candidate data with expected completeness less than a completeness threshold from each disk data.

[0053] The integrity threshold represents a pre-configured threshold for expected integrity. The size of the integrity threshold may be related to the computing power of the disk to be tested. For example, if the computing power of the disk to be tested is greater, the amount of data that can be processed per unit time is greater, and therefore, the expected integrity can be greater.

[0054] Specifically, the test terminal can determine the computing capability of the disk to be tested. The test terminal can determine the integrity threshold corresponding to the disk to be tested according to the computing capability. The test terminal can screen multiple candidate data with expected integrity less than the integrity threshold from each disk data.

[0055] In one embodiment, the computing capacity of the disk to be tested may include the computing capacity of the disk to be tested and the computing capacity of the terminal to which the disk to be tested belongs. Specifically, the inspection terminal determines the computing capacity of the disk to be tested and the computing capacity of the terminal to which the disk to be tested belongs. The inspection terminal determines the integrity threshold corresponding to the disk to be tested based on the two computing capacities.

[0056] In one embodiment, the computing capability to be tested may also include the computing capability of the test terminal. Specifically, the test terminal determines the computing capability of the disk to be tested, the computing capability of the terminal to which the disk to be tested belongs, and the computing capability of the test terminal itself. The test terminal determines the integrity threshold corresponding to the disk to be tested based on these three computing capabilities.

[0057] In one embodiment, after obtaining multiple computing capabilities, the inspection terminal can determine the smaller computing capability from the two (or three) computing capabilities as the minimum computing capability of the disk to be tested. The inspection terminal can determine the integrity threshold corresponding to the disk to be tested based on the minimum computing capability. The minimum computing capability and the integrity threshold are positively correlated.

[0058] S208, performing a cyclic redundancy check on each candidate data to obtain a check result of the candidate data.

[0059] Among them, cyclic redundancy check can refer to a fast algorithm that generates a short fixed-bit check code based on network data packets or computer files. This algorithm can be used to detect or check errors that may occur after data transmission or storage, that is, it can be used to check the integrity of disk data.

[0060] The verification result may indicate the result of the cyclic redundancy check in the candidate data. The verification result may include a cyclic redundancy check failure and a cyclic redundancy check pass. A cyclic redundancy check failure indicates that the data integrity of the candidate data has not passed, and the candidate data may be reported to facilitate subsequent data maintenance personnel to perform corresponding data recovery operations. A cyclic redundancy check pass indicates that the data integrity of the candidate data has passed the verification.

[0061] The cyclic redundancy check includes initializing the CRC check module, calculating the CRC value, storing the CRC value, reading the data and checking. Specifically, the check terminal can be configured with a CRC module. The check terminal can calculate the CRC calculation value of the candidate data through the CRC algorithm. The check terminal can compare the CRC acquisition value of the obtained candidate data and the CRC calculation value. If they are the same, it means that no error has occurred in the candidate data during transmission or storage, and the check result of the candidate data is passed; otherwise, it means that the data may have been damaged, and the check result of the candidate data is not passed.

[0062] S210, determining target data whose verification result fails from each candidate data.

[0063] The target data refers to candidate data that has failed the verification result. The target data may also refer to data with low data integrity.

[0064] The target data can be repaired. Specifically, after obtaining the target data, the inspection terminal can take measures such as rewriting the data or using backup data for repair.

[0065] In the above-mentioned method for screening disk data, the disk attribute information of the disk to be tested is obtained, and the disk attribute information includes the disk status information of the disk to be tested, and the data attribute information of each of the multiple disk data stored on the disk to be tested. According to the disk status information and each data attribute information, the data integrity analysis is performed on each disk data to obtain the expected integrity of each disk data. The expected integrity reflects the estimated integrity of each disk data, which is conducive to the subsequent screening of each data and improves the verification efficiency of the disk data. Using the above-mentioned method, in each disk data, multiple candidate data with expected integrity less than the integrity threshold are screened, each disk data is first preliminarily screened, and then for each candidate data, a cyclic redundancy check is performed on the candidate data to obtain the verification result of the candidate data, and the target data whose verification result does not pass is determined from each candidate data, thereby reducing the amount of data for the cyclic redundancy check and improving the verification efficiency.

[0066] In one embodiment, data integrity analysis is performed on each disk data according to disk status information and data attribute information to obtain the expected integrity of each disk data, including: feature extraction of disk status information to obtain disk status features of the disk to be tested, feature extraction of data attribute information to obtain data attribute features of the disk data, inputting the disk status features and data attribute features into an integrity analysis model, and determining the predicted integrity of the disk data based on the output of the integrity analysis model.

[0067] The disk status feature may represent a feature corresponding to the disk status information that can be directly processed by the neural network model. The disk status feature may include the SMART status feature of the disk to be tested, the data corruption feature of the disk to be tested, and the like.

[0068] The data attribute features may represent features corresponding to data attribute information that can be directly processed by the neural network model. The data attribute features may include data size features, write time features, historical read count features, etc.

[0069] The completeness analysis model may be an analysis model pre-trained for disk data and used to determine the predicted completeness of disk data. The completeness analysis model may be a pre-trained reinforcement learning model.

[0070] The output of the integrity analysis model may represent the direct output result of the integrity analysis model.

[0071] Specifically, the inspection terminal may determine the predicted integrity of the disk data based on the output of the integrity analysis model.

[0072] In one embodiment, the output of the integrity analysis model may include the integrity of the disk data and the confidence corresponding to the integrity. The inspection terminal may obtain a confidence threshold. The inspection terminal determines the integrity corresponding to the confidence greater than the confidence threshold as the predicted integrity of the disk data.

[0073] In one embodiment, when the confidence levels of multiple integrity levels of the disk to be tested are all less than a confidence threshold, the inspection terminal may determine the integrity level corresponding to the maximum confidence level among the confidence levels as the predicted integrity level of the disk data.

[0074] There are many ways to extract features from disk status information. For example, the inspection terminal can extract features from disk status information according to feature extraction parameters in the feature extraction module to obtain disk status features of the disk to be tested. There are many ways to extract features from data attribute information. For example, the inspection terminal can extract features from data attribute information according to feature extraction parameters in the feature extraction module to obtain data attribute features of disk data.

[0075] In one of the embodiments, when performing feature extraction, the inspection terminal may pre-process the information. For example, the inspection terminal may delete redundant information or merge repeated information.

[0076] In this embodiment, feature extraction is performed on the disk status information to obtain the disk status features of the disk to be tested, and feature extraction is performed on the data attribute information to obtain the data attribute features of the disk data. The disk status features and the data attribute features are input into the integrity analysis model, and the predicted integrity of the disk data is determined based on the output of the integrity analysis model. Feature extraction is performed on the disk status information and the data attribute information, which facilitates the integrity analysis model to predict the integrity of the disk data and can improve the accuracy of the predicted integrity.

[0077] In one embodiment, as shown in FIG3 , which is a flow chart of a method for training a completeness analysis model, the training process of the completeness analysis model includes:

[0078] S302, obtaining an initial model and a plurality of disk data samples carrying integrity labels.

[0079] Among them, as shown in the structural diagram of the initial model in Figure 4, the initial model can be an analysis model to be trained. The initial model can include an environment module and an agent module. The environment module can be an external system for the agent module to learn and make decisions.

[0080] The intelligent agent module may be a module constructed using the DDPG algorithm, and the intelligent agent module is used to generate a review score.

[0081] The agent module includes a strategy submodule and a judgment submodule. The strategy submodule includes a strategy network and a target strategy network. The strategy network receives the current state as input and outputs an action, i.e., a data score. The target strategy network receives the next state as input and outputs the action of the next state.

[0082] The evaluation submodule is used to calculate the Q value. The evaluation submodule includes the evaluation network and the target evaluation network. The evaluation network receives the current state as input, and also receives the action generated by the policy network as an additional input, and the value function under the given data state and action. The target evaluation network receives the next state as input, and also receives the next state action generated by the policy network as an additional input.

[0083] In order to simulate the disk environment, as shown in the structural diagram of the state space in FIG5 , a dynamic state space can be set to dynamically update the data state as disk data is added and existing data changes.

[0084] The disk data sample is the sample to be trained for the initial model. The disk data sample includes the disk data and the label corresponding to the disk data, that is, the completeness label.

[0085] The integrity tag indicates the integrity of the disk data in each disk data sample.

[0086] Specifically, the inspection terminal can collect multiple initial data samples. The data obtained by the inspection terminal includes damaged data and undamaged data, and determines the integrity label of each initial sample data. The inspection terminal can obtain a disk data sample carrying an integrity label based on the integrity label and each initial sample data. Among them, the integrity label corresponding to the damaged initial data sample is incomplete and the data integrity is lower than a threshold. The integrity label corresponding to the undamaged initial data sample is complete and the data integrity is higher than a threshold.

[0087] S304: Perform model training on the initial model based on each disk data sample to obtain the prediction completeness of each disk data sample.

[0088] The prediction completeness may represent the output result of the initial model for disk data during the current round of iterations.

[0089] The process of model training can be the process of updating the parameters in the environment module and the agent module.

[0090] S306, determining the actual completeness ranking and the predicted completeness ranking of each disk data sample based on the completeness label and predicted completeness of each disk data sample.

[0091] The actual integrity ranking reflects the ranking result between the integrity labels of each disk data. Specifically, the integrity label of the disk data ranked higher has a greater integrity than the integrity label of the disk data ranked lower.

[0092] Similarly, the predicted completeness ranking reflects the ranking result between the predicted completeness of each disk data. Specifically, the predicted completeness of the disk data ranked higher is greater than the predicted completeness of the disk data ranked lower.

[0093] The method for determining the actual completeness ranking and predicted completeness ranking of each disk data sample can be descending sorting or ascending sorting, which can be flexibly selected according to the actual situation and is not limited here.

[0094] S308, determining the difference between the actual completeness ranking and the predicted completeness ranking, and returning to the step of training the initial model based on each disk data sample to perform the next round of iteration.

[0095] Among them, the difference between the actual completeness ranking and the predicted completeness ranking can represent the overall difference between the two ranking results. The larger the difference, the lower the accuracy of the initial model. Conversely, the smaller the difference, the higher the accuracy of the initial model.

[0096] Specifically, the inspection terminal may directly return to the step of performing model training on the initial model based on each disk data sample.

[0097] Specifically, the test terminal may first perform sample update on each disk data sample to obtain an updated sample, and the test terminal may return to the step of performing model training on the initial model based on the updated sample.

[0098] S310, when the training end condition is met, a completeness analysis model is obtained.

[0099] The training end condition may be an end condition configured for the training process of the initial model.

[0100] The training end conditions may include that the number of model iterations is greater than a preset value, the difference of the model is less than a preset value, and the change in the difference of the model meets the preset conditions. For example, if the difference of the model in round X is 0.1, and the difference of the model in round X+1 is 0.11, it can be determined that the change in the difference of the model in round X meets the preset conditions, and the model after the Xth round of iteration is determined as the integrity analysis model.

[0101] In this embodiment, the difference between the actual completeness ranking and the predicted completeness ranking is determined, and the step of training the initial model based on each disk data sample is returned to perform the next round of iteration. When the training end condition is met, the completeness analysis model is obtained, the initial model is trained, and the initial model is trained based on the difference between the actual completeness ranking and the predicted completeness ranking. The initial model is trained taking into account the ranking of each disk data sample in each disk data sample, so that the trained model has the ability to perceive the expected completeness of each disk data, thereby improving the analysis accuracy of the completeness analysis model.

[0102] In one embodiment, when the training end conditions are met, a completeness analysis model is obtained, including: obtaining the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round, obtaining the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round, and when the coefficient of variation of the difference between the two rounds within the preset rounds is less than a preset value, that is, the model is stable, and the completeness analysis model can be obtained.

[0103] The historical difference may represent the difference between the actual completeness ranking and the predicted completeness ranking in the previous round.

[0104] During the model training process, the difference can be gradually reduced as the number of iterations increases.

[0105] If the coefficient of variation of the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round is less than a preset value within the preset round, it means that the model of the current iteration round is close to fitting or has been fitted, and the model of the current iteration round can be determined as the completeness analysis model.

[0106] Specifically, the verification terminal can obtain the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round. The verification terminal can determine the model of the current iteration round as the completeness analysis model if the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round has a coefficient of variation within a preset round that is less than a preset value. The verification terminal can also return to the step of training the initial model based on each disk data sample and perform the next round of iteration if the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the historical difference in the previous round has a coefficient of variation within a preset round that is greater than a preset value.

[0107] In this embodiment, the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round is obtained. When the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the historical difference has a coefficient of variation less than a preset value within a preset round, a completeness analysis model is obtained. The numerical relationship between the difference and the historical difference is used to accurately determine whether the model is fitted or close to fitting, thereby accurately obtaining the completeness analysis model.

[0108] In one embodiment, the method for screening disk data further includes: obtaining the computing capability corresponding to the disk to be tested, and determining a completeness threshold matching the computing capability, wherein the completeness threshold is positively correlated with the computing capability.

[0109] The computing power corresponding to the disk to be tested can represent the disk read and write capabilities of the disk to be tested, such as IOPS and throughput. The number of input and output operations per second (IOPS) can represent the number of read and write operations that the disk can handle per unit time. Throughput can represent the amount of data transferred per second by the disk, usually in MB / s (megabytes per second). Throughput reflects the performance of the disk in sequential read and write operations, such as the speed of copying large files or large compressed packages.

[0110] The computing capability corresponding to the disk to be tested may also represent the computing capability of the terminal to which the disk to be tested belongs, or the computing capability of the test terminal, for example, the number of instructions per second, the number of clock cycles per instruction, the throughput, the CPU frequency, the storage capacity, the bandwidth, etc. of the terminal to which the disk to be tested belongs.

[0111] The higher the computing power, the more disk data can be screened per unit time, and the corresponding integrity threshold can be higher. While ensuring the extraction efficiency, the number of candidate data items can be increased to further improve the accuracy of screening and the accuracy of data integrity verification.

[0112] In this embodiment, the computing power corresponding to the disk to be tested is obtained, and the integrity threshold that matches the computing power is determined. The integrity threshold is positively correlated with the computing power. The integrity threshold and the computing power are matched. While ensuring the extraction efficiency, the number of candidate data items is increased, and the screening accuracy and the accuracy of data integrity verification are further improved.

[0113] In one embodiment, obtaining the computing capacity corresponding to the disk to be tested includes: determining a terminal used to process the disk to be tested, obtaining the disk computing capacity of the disk to be tested and the terminal computing capacity of the terminal, and determining the computing capacity corresponding to the disk to be tested based on the disk computing capacity and the terminal computing capacity.

[0114] The terminal used for processing the disk to be tested may be a terminal to which the disk to be tested belongs, or may be a testing terminal.

[0115] The disk computing capability of the disk to be tested may represent the disk read and write capability of the disk to be tested, such as IOPS and throughput.

[0116] The terminal computing capability may represent the computing capability of the terminal to which the disk to be tested belongs or the computing capability of the test terminal. Taking the computing capability of the test terminal as an example, the computing capability of the test terminal includes the number of instructions per second, the number of clock cycles per instruction, throughput, CPU frequency, storage capacity, bandwidth, etc. of the test terminal.

[0117] Instructions per second (IPS) indicates the number of instructions that the CPU can execute per second. Clock cycles per instruction (CPI) indicates the average number of clock cycles required to execute an instruction. The lower the CPI value, the faster the CPU executes instructions. Throughput indicates the amount of information that the test terminal can process within a certain time interval, which is usually closely related to CPU consumption, external interfaces, and IO. CPU frequency indicates the CPU main frequency of the test terminal. Storage capacity indicates the size of the storage space in the test terminal. Bandwidth indicates the theoretical performance peak of the system main memory, which is closely related to computing power.

[0118] Specifically, after the inspection terminal obtains the disk computing capacity and the terminal computing capacity, the smaller computing capacity may be determined as the computing capacity corresponding to the disk to be tested.

[0119] In one embodiment, the inspection terminal may determine the computing capacity corresponding to the IOPS and the computing capacity corresponding to the throughput according to the IOPS and the throughput. The inspection terminal determines the smaller computing capacity as the disk computing capacity.

[0120] In one embodiment, the inspection terminal may determine the computing power corresponding to the number of instructions per second, the computing power corresponding to the number of clock cycles per instruction, the computing power corresponding to the throughput, the computing power corresponding to the CPU frequency, the computing power corresponding to the storage capacity, and the computing power corresponding to the bandwidth according to the number of instructions per second, the number of clock cycles per instruction, the throughput, the CPU frequency, the storage capacity, and the bandwidth of the inspection terminal. The inspection terminal determines the smaller computing power as the terminal computing power.

[0121] In one of the embodiments, the inspection terminal may also determine the computing capacity corresponding to the disk to be tested according to the computing capacity corresponding to the terminal to which the disk to be tested belongs, the computing capacity of the disk, and the computing capacity of the inspection terminal.

[0122] In this embodiment, a terminal used to process the disk to be tested is determined, the disk computing capacity of the disk to be tested and the terminal computing capacity of the terminal are obtained, and the computing capacity corresponding to the disk to be tested is determined based on the disk computing capacity and the terminal computing capacity. Determining the accurate computing capacity corresponding to the disk to be tested is conducive to determining an accurate integrity threshold, thereby improving the accuracy of data integrity verification.

[0123] In one embodiment, the disk data review method based on artificial intelligence as shown in FIG6 includes:

[0124] The first part, model training, includes:

[0125] S602, obtaining an initial model and a plurality of disk data samples carrying integrity labels.

[0126] Exemplarily, as shown in Figure 7, the artificial intelligence disk data review method includes step 1, data collection and preprocessing; step 2, establishing a disk data review system model; step 3, disk data review system model training; step 4, data update; step 5, model continuous learning; step 6, output data sorting results.

[0127] Specifically, step 1: data collection and preprocessing, real-time monitoring of data from multiple disks, as shown in the structural diagram of the data collection and preprocessing module in Figure 8, collects and performs feature screening, and sets training sets and validation sets.

[0128] Step 1 includes step 1.1 and step 1.2. Among them, step 1.1, real-time monitoring and collection of disk data information within a period of time, including: obtaining the SMART status of each disk and the disk data damage rate; collecting various disk data including data size, write time, latest read time, read times, last review time, whether the data is deleted, the user level and other attribute characteristics as the disk data status information.

[0129] Step 1.2 includes preprocessing the acquired information. According to the time of data corruption, the damaged data score is set, the data close to corruption or already corrupted is set with a high review score, and the uncorrupted data is set with a low review score. The machine learning model random forest is used to evaluate the importance of features for feature screening.

[0130] Step 2: Establish a disk data review system model, as shown in Figure 4, including an environment module and an agent module. Step 2 includes step 2.1 and step 2.2.

[0131] Among them, the environment module in step 2.1 is the external system for the agent module to learn and make decisions. It consists of state space, action space, reward mechanism, and termination condition. Step 2.1 includes steps 2.1.1 to 2.1.4.

[0132] Among them, step 2.1.1, the state space of the environment module: the state is the representation of the environment at the current time point, and the state information of the preprocessed data is represented by a vector, where each data feature is an element in the vector. The data in the disk is constantly written and deleted. In order to more accurately simulate the disk environment, a dynamic state space is set here to dynamically update the data state as the disk data is added and the existing data changes.

[0133] Step 2.1.2 Wherein the environment module action space: The action is defined as scoring each data. This means that the policy module needs to learn how to generate a score based on the current data state.

[0134] Step 2.1.3 The environment module reward mechanism is a key part of guiding the agent. When the agent's ranking prediction matches the actual result, a positive reward is given. If the prediction error is large, a negative reward is given. When the score given by the agent for the data is close to the actual result, an additional positive reward is given.

[0135] Step 2.1.4 The environment module terminates when a predetermined number of iterations is reached or when the agent's prediction performance stops improving on the validation set.

[0136] Step 2.2: The agent module is constructed using the DDPG algorithm. The agent module includes a strategy submodule and a judgment submodule.

[0137] Step 2.2.1 The policy submodule is used to generate the review score. The policy submodule includes a policy network and a target policy network. The policy network receives the current state S as input and outputs an action, i.e., the disk data score. The target policy network receives the next state S' as input and outputs the action of the next state.

[0138] Step 2.2.2 The evaluation submodule is used to calculate the Q value. The evaluation submodule includes a judgment network and a target evaluation network. The judgment network receives the current state S as input, and also receives the action generated by the policy network as an additional input, and outputs the value function under the data state and action. The target evaluation network receives the next state S' as input, and also receives the next state action generated by the policy network as an additional input. Among them, the Q value can represent the expected reward of the current state.

[0139] Step 3: Disk data review system model training: Use priority experience replay technology to prioritize the data in the experience area, and update the data review list by training the agent multiple times in a row.

[0140] Step 3.1: Initialize the parameters of the policy network and the critic network. Initialize the experience replay buffer to store the experience of the agent interacting with the environment.

[0141] Step 3.2: Collect states. In the training set, use the current policy network to select an action A based on the current state S at each step. Execute action A and interact with the environment to return a new state S' and reward R. Store the experience (S, A, R, S') in the experience replay buffer and calculate a priority for each experience based on the TD error. The TD error can represent the difference between the reward of the current state and the expected reward of the next state, that is, the difference between the Q value of the current state and the Q value of the next state.

[0142] Step 3.3: Training, sample a batch of experience data from the experience replay buffer according to the priority. Use the judgment network to calculate the Q value of these experience data and calculate the target Q value. Use the mean square error to calculate the loss of the judgment network. Update the parameters of the judgment network through the back propagation algorithm. Use the Q value gradient of the judgment network to update the parameters of the policy network to maximize the expected return. Among them, the sampling can be uniformly spaced sampling, for example, sampling every two data in sequence.

[0143] Step 3.4: Update the target network. Every certain number of training iterations, use the parameters of the current policy network and the judgment network to update the parameters of the target policy network and the target judgment network. Through the soft update method, the parameters of the current network are updated to the target network at a ratio of 0.001.

[0144] Step 3.5: Iterate and repeat steps 2-4 until the termination condition is reached.

[0145] Step 4: Data update: The training set involves the addition of data and changes in the state of existing data, and the state space of the environment model is dynamically updated.

[0146] Step 4.1: Data Addition: When new data is added, the state space needs to be expanded accordingly to include the features of the new data.

[0147] Step 4.2: Data update: At fixed time intervals, update the state space of the environment module according to data changes. When data is deleted: set the data to be deleted in the state space. Other data needs to update the SMART status of each disk, disk data corruption rate, data size, latest read time, number of reads, last review time, whether the data is deleted, user level and other attribute characteristics.

[0148] Step 5: The model continues to learn. When the state space changes, the policy network and the judgment network parameters are updated according to the new state space to find a new optimal strategy. This process is iterative until the termination condition is reached.

[0149] Step 5.1: Update the state, use the updated state space, and use the current policy network to select an action A based on the current state S at each step. Execute action A, and interact with the environment to return the new state S' and reward R. Store the experience (S, A, R, S') in the experience replay buffer, and calculate a priority for each experience based on the TD error.

[0150] Step 5.2: Training, sample a batch of experience data from the experience replay buffer according to the priority. Use the critic network to calculate the Q value of these experience data and calculate the target Q value. Use the mean square error to calculate the loss of the critic network. Update the parameters of the critic network through the back-propagation algorithm. Use the Q value gradient of the critic network to update the parameters of the policy network to maximize the expected return.

[0151] Step 5.3: Update the target network. Every certain number of iterations, use the parameters of the policy network and the judgment network to update the parameters of the target policy network and the target judgment network. Through the soft update method, the parameters of the network are updated to the target network at a ratio of 0.001.

[0152] Step 5.4: Iterate and repeat Step 5.1 to Step 5.4 until the termination condition is reached.

[0153] Step 6: Output data sorting results: Validate the trained agent model on the validation set. Eventually, the agent will be able to score and sort based on the input data state, and data sorting is the optimal review method.

[0154] S604: Perform model training on the initial model based on each disk data sample to obtain the prediction completeness of each disk data sample.

[0155] S606, determining the actual completeness ranking and the predicted completeness ranking of each disk data sample based on the completeness label and predicted completeness of each disk data sample.

[0156] S608: Determine the difference between the actual completeness ranking and the predicted completeness ranking.

[0157] S610, obtaining a historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round.

[0158] S612, determine whether the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference with the historical difference satisfies that the coefficient of variation within the preset round is less than the preset value. If so, execute S614, if not, return to S604 and increase the number of iterations by one.

[0159] The difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round can be expressed as the current difference.

[0160] S614, determining the initial model in the current iteration round as the integrity analysis model.

[0161] The second part, model application, includes:

[0162] S616, determining a terminal for processing the disk to be tested.

[0163] S618, obtaining the disk computing capability of the disk to be tested and the terminal computing capability of the terminal.

[0164] S620: Determine the computing capability corresponding to the disk to be tested based on the computing capability of the disk and the computing capability of the terminal.

[0165] S622, determine a completeness threshold that matches the computing capability.

[0166] Among them, the completeness threshold is positively correlated with computing power.

[0167] S624, obtaining disk attribute information of the disk to be tested.

[0168] The disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested.

[0169] S626, extracting features from the disk status information to obtain disk status features of the disk to be tested.

[0170] S628, extracting features from the data attribute information to obtain data attribute features of the disk data.

[0171] S630, inputting the disk status characteristics and the data attribute characteristics into the integrity analysis model, and determining the predicted integrity of the disk data based on the output of the integrity analysis model.

[0172] S632: Filter multiple candidate data whose expected completeness is less than a completeness threshold from each disk data.

[0173] S634, for each candidate data, perform a cyclic redundancy check on the candidate data to obtain a check result of the candidate data.

[0174] S636, determining the target data that fails the verification result from each candidate data.

[0175] In this embodiment, the disk attribute information of the disk to be tested is obtained, and the disk attribute information includes the disk status information of the disk to be tested and the data attribute information of each of the multiple disk data stored in the disk to be tested. According to the disk status information and the attribute information of each data, the data integrity analysis is performed on each disk data to obtain the expected integrity of each disk data. The expected integrity reflects the estimated integrity of each disk data, which is conducive to the subsequent screening of each data and improves the verification efficiency of the disk data. Using the above method, in each disk data, multiple candidate data with expected integrity less than the integrity threshold are screened, each disk data is first preliminarily screened, and then for each candidate data, a cyclic redundancy check is performed on the candidate data to obtain the verification result of the candidate data, and the target data whose verification result does not pass is determined from each candidate data, thereby reducing the amount of data for the cyclic redundancy check and improving the verification efficiency.

[0176] The above method can also bring the following beneficial effects:

[0177] 1. The application formulates the optimal review method by assigning reasonable review scores to the data. This method not only improves the review efficiency, but also makes reasonable allocation of system resources, making the review work more efficient and energy-saving.

[0178] 2. The application uses artificial intelligence technology to automatically discover damaged or potentially damaged data. Compared with traditional methods, it can more accurately identify potential problems, so that timely measures can be taken to avoid data loss or damage.

[0179] 3. The application performs fine-grained monitoring of multiple disk data, and can perform fine-grained monitoring based on multiple characteristics of the data and the SMART status of the disk.

[0180] 4. The application uses continuous learning technology to track the dynamic changes of multiple disk data in real time and dynamically adjust the model accordingly, which not only quickly adapts to data changes but also retains the memory of past learning results. This innovation makes the model more accurate and efficient when formulating disk data review optimization methods.

[0181] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0182] Based on the same inventive concept, the embodiment of the present application also provides a disk data screening device for implementing the above-mentioned disk data screening method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the one or more disk data screening device embodiments provided below can refer to the limitations of the disk data screening method above, and will not be repeated here.

[0183] In one embodiment, as shown in FIG. 9 , a disk data screening device is provided, including: an acquisition module 902, an analysis module 904, a screening module 906, a verification module 908 and a processing module 910, wherein:

[0184] An acquisition module 902 is used to acquire disk attribute information of the disk to be tested, where the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested;

[0185] The analysis module 904 is used to perform data integrity analysis on each disk data according to the disk status information and each data attribute information, and obtain the expected integrity of each disk data;

[0186] A screening module 906 is used to screen multiple candidate data with expected completeness less than a completeness threshold from each disk data;

[0187] A verification module 908 is used to perform a cyclic redundancy check on each candidate data to obtain a verification result of the candidate data;

[0188] The processing module 910 is used to determine the target data whose verification result fails from each candidate data.

[0189] In one embodiment, the analysis module 904 is also used to extract features from the disk status information to obtain disk status features of the disk to be tested; to extract features from the data attribute information to obtain data attribute features of the disk data; to input the disk status features and the data attribute features into the integrity analysis model, and to determine the predicted integrity of the disk data based on the output of the integrity analysis model.

[0190] In one embodiment, the analysis module 904 is also used to obtain an initial model and multiple disk data samples carrying integrity labels; perform model training on the initial model based on each disk data sample to obtain the predicted integrity of each disk data sample; determine the actual integrity ranking and predicted integrity ranking of each disk data sample based on the integrity label and predicted integrity of each disk data sample; determine the difference between the actual integrity ranking and the predicted integrity ranking, and return to the step of performing model training on the initial model based on each disk data sample to perform the next round of iteration; when the training end conditions are met, obtain the integrity analysis model.

[0191] In one embodiment, the analysis module 904 is also used to obtain the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round; when the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the historical difference satisfies the condition that the coefficient of variation within the preset round is less than a preset value, a completeness analysis model is obtained.

[0192] In one of the embodiments, the disk data screening device further includes: a threshold determination module for obtaining the computing capability corresponding to the disk to be tested; determining a completeness threshold matching the computing capability; and the completeness threshold is positively correlated with the computing capability.

[0193] In one of the embodiments, the threshold determination module is also used to determine the terminal used to process the disk to be tested; obtain the disk computing capacity of the disk to be tested and the terminal computing capacity of the terminal; and determine the computing capacity corresponding to the disk to be tested based on the disk computing capacity and the terminal computing capacity.

[0194] Each module in the above-mentioned disk data screening device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0195] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store disk attribute information data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for screening disk data is implemented.

[0196] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0197] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method steps are implemented when the processor executes the computer program.

[0198] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0199] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for screening disk data, characterized in that: The method comprises: Obtaining disk attribute information of the disk to be tested, wherein the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested; According to the disk status information and the attribute information of each data, data integrity analysis is performed on each disk data to obtain the expected integrity of each disk data; In each of the disk data, a plurality of candidate data with the expected completeness being less than a completeness threshold are screened; For each of the candidate data, performing a cyclic redundancy check on the candidate data to obtain a check result of the candidate data; Target data whose verification result fails is determined from each of the candidate data.

2. The method according to claim 1, characterized in that The step of performing data integrity analysis on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data includes: Extracting features from the disk status information to obtain disk status features of the disk to be tested; Extracting features from the data attribute information to obtain data attribute features of the disk data; The disk status characteristics and the data attribute characteristics are input into an integrity analysis model, and based on the output of the integrity analysis model, the predicted integrity of the disk data is determined.

3. The method according to claim 2, characterized in that The training process of the integrity analysis model includes: Obtain an initial model and multiple disk data samples with integrity labels; Performing model training on the initial model based on each of the disk data samples to obtain a prediction completeness of each of the disk data samples; Determining an actual completeness ranking and a predicted completeness ranking of each of the disk data samples based on the completeness label and predicted completeness of each of the disk data samples; Determine the difference between the actual completeness ranking and the predicted completeness ranking, and return to the step of training the initial model based on each of the disk data samples to perform the next round of iteration; When the training end conditions are met, a completeness analysis model is obtained.

4. The method according to claim 3, characterized in that When the training end condition is met, obtaining the integrity analysis model includes: Obtaining a historical difference between the actual completeness ranking and the predicted completeness ranking in a previous round of the current iteration round; When the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the historical difference has a coefficient of variation within a preset round that is less than a preset value, a completeness analysis model is obtained.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining the computing capacity corresponding to the disk to be tested; Determine a completeness threshold that matches the computing capability; the completeness threshold is positively correlated with the computing capability.

6. The method according to claim 5, characterized in that The obtaining of the computing capability corresponding to the disk to be tested includes: Determine a terminal for processing the disk to be tested; Acquire the disk computing capability of the disk to be tested and the terminal computing capability of the terminal; Based on the disk computing capability and the terminal computing capability, the computing capability corresponding to the disk to be tested is determined.

7. A disk data screening device, characterized in that: The device comprises: An acquisition module, used for acquiring disk attribute information of the disk to be tested, wherein the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested; An analysis module, configured to perform data integrity analysis on each disk data according to the disk status information and each data attribute information, and obtain the expected integrity of each disk data; A screening module, used for screening, from each of the disk data, a plurality of candidate data whose expected completeness is less than a completeness threshold; A verification module, configured to perform a cyclic redundancy check on each candidate data to obtain a verification result of the candidate data; The processing module is used to determine the target data whose verification result fails from each of the candidate data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images