Method and device for screening disk data, computer device and storage medium

Abstract

The application relates to a disk data screening method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring disk attribute information of a to-be-tested disk, the disk attribute information comprising disk state information of the to-be-tested disk and data attribute information of a plurality of disk data stored in the to-be-tested disk; performing data integrity analysis on the disk data according to the disk state information and the data attribute information, obtaining expected integrity of each disk data; and screening a plurality of candidate data with expected integrity less than a threshold value from the disk data by using the above method. The candidate data is screened from the disk data, and the cyclic redundancy check is performed on each candidate data, so that the check result of the candidate data is obtained, the target data with a failed check result is determined from the candidate data, the data amount of the cyclic redundancy check is reduced, and the check efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of disk inspection, in particular to a disk data screening method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] Disks are widely used in scenarios involving data storage. The stored data in the disks is crucial to an enterprise. Once the data in the disks is damaged or lost, it may cause serious consequences such as business interruption, information leakage or economic loss. In order to ensure the accuracy and reliability of the stored data, data integrity verification of the stored data in the disks is needed.

[0003] In the traditional technology, the integrity of each stored data in the disks is verified in sequence to obtain the integrity of each stored data, which has the problem of low verification efficiency. SUMMARY

[0004] Therefore, it is necessary to provide a disk data screening method, device, computer equipment, computer readable storage medium and computer program product capable of improving verification efficiency in view of the above technical problems.

[0005] In a first aspect, the present application provides a disk data screening method. The method comprises:

[0006] obtaining disk attribute information of a to-be-tested disk, the disk attribute information comprising disk state information of the to-be-tested disk and data attribute information of a plurality of disk data stored in the to-be-tested disk; performing data integrity analysis on each of the disk data according to the disk state information and each of the data attribute information to obtain an expected integrity of each of the disk data; screening a plurality of candidate data with an expected integrity less than an integrity threshold from the disk data; performing a cyclic redundancy check on each of the candidate data to obtain a check result of the candidate data; and determining a target data with a failed check result from the candidate data.

[0007] In one of the embodiments, the performing data integrity analysis on each of the disk data according to the disk state information and each of the data attribute information to obtain an expected integrity of each of the disk data comprises: performing feature extraction on the disk state information to obtain disk state features of the to-be-tested disk; performing feature extraction on the data attribute information to obtain data attribute features of the disk data; inputting the disk state features and the data attribute features into an integrity analysis model, and determining a predicted integrity of the disk data based on an output of the integrity analysis model.

[0008] In one of the embodiments, the training process of the completeness analysis model comprises: obtaining an initial model and a plurality of disk data samples carrying completeness labels; performing model training on the initial model based on each of the disk data samples to obtain a predicted completeness of each of the disk data samples; determining an actual completeness ranking and a predicted completeness ranking of each of the disk data samples based on the completeness label and the predicted completeness of each of the disk data samples; determining a difference between the actual completeness ranking and the predicted completeness ranking, and returning to the step of performing model training on the initial model based on each of the disk data samples for the next iteration; and obtaining the completeness analysis model when a training end condition is met.

[0009] In one of the embodiments, obtaining the completeness analysis model when the training end condition is met comprises: obtaining a historical difference between the actual completeness ranking and the predicted completeness ranking in the previous iteration of the current iteration; and obtaining the completeness analysis model when a difference between the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration and the historical difference is less than a preset value within a preset number of iterations.

[0010] In one of the embodiments, the method further comprises: obtaining a computing capability corresponding to the disk under test; determining a completeness threshold matching the computing capability; and the completeness threshold is positively correlated with the computing capability.

[0011] In one of the embodiments, obtaining the computing capability corresponding to the disk under test comprises: determining a terminal used for processing the disk under test; obtaining a disk computing capability of the disk under test and a terminal computing capability of the terminal; and determining the computing capability corresponding to the disk under test based on the disk computing capability and the terminal computing capability.

[0012] In a second aspect, the application further provides a disk data screening device. The device comprises:

[0013] The obtaining module is configured to obtain disk attribute information of a disk under test, the disk attribute information comprising disk state information of the disk under test and data attribute information of a plurality of disk data stored in the disk under test; the analysis module is configured to perform data completeness analysis on each of the disk data according to the disk state information and each of the data attribute information to obtain an expected completeness of each of the disk data; the screening module is configured to screen a plurality of candidate data with an expected completeness less than a completeness threshold from each of the disk data; the verification module is configured to perform a cyclic redundancy check on each of the candidate data to obtain a verification result of each of the candidate data; and the processing module is configured to determine target data with a failed verification result from each of the candidate data.

[0014] In a third aspect, the present application provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor implements the following steps when executing the computer program:

[0015] obtaining disk attribute information of a to-be-tested disk, the disk attribute information comprising disk state information of the to-be-tested disk and data attribute information of each of a plurality of disk data stored in the to-be-tested disk; performing data integrity analysis on each of the disk data according to the disk state information and the data attribute information of each of the disk data, to obtain an expected integrity of each of the disk data; screening a plurality of candidate data from the disk data, the expected integrity of each of the candidate data being less than an integrity threshold; performing cyclic redundancy check on each of the candidate data, to obtain a check result of each of the candidate data; and determining a target data from the candidate data, the check result of the target data being failed.

[0016] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0017] obtaining disk attribute information of a to-be-tested disk, the disk attribute information comprising disk state information of the to-be-tested disk and data attribute information of each of a plurality of disk data stored in the to-be-tested disk; performing data integrity analysis on each of the disk data according to the disk state information and the data attribute information of each of the disk data, to obtain an expected integrity of each of the disk data; screening a plurality of candidate data from the disk data, the expected integrity of each of the candidate data being less than an integrity threshold; performing cyclic redundancy check on each of the candidate data, to obtain a check result of each of the candidate data; and determining a target data from the candidate data, the check result of the target data being failed.

[0018] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program. The computer program is executed by a processor to implement the following steps:

[0019] Obtaining disk attribute information of a to-be-tested disk, the disk attribute information including disk state information of the to-be-tested disk and data attribute information of each of a plurality of disk data stored in the to-be-tested disk; performing data integrity analysis on each of the disk data according to the disk state information and the data attribute information of each of the disk data, to obtain an expected integrity of each of the disk data; screening, from the disk data, a plurality of candidate data whose expected integrity is less than an integrity threshold; performing a cyclic redundancy check on each of the candidate data, to obtain a check result of the candidate data; and determining, from the candidate data, a target data whose check result fails.

[0020] The method, the device, the computer device, the storage medium and the computer program product for screening the disk data, obtain disk attribute information of a to-be-tested disk, the disk attribute information including disk state information of the to-be-tested disk and data attribute information of each of a plurality of disk data stored in the to-be-tested disk; perform data integrity analysis on each of the disk data according to the disk state information and the data attribute information of each of the disk data, to obtain an expected integrity of each of the disk data; screen, from the disk data, a plurality of candidate data whose expected integrity is less than an integrity threshold; perform a cyclic redundancy check on each of the candidate data, to obtain a check result of the candidate data; and determine, from the candidate data, a target data whose check result fails. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 An application environment diagram of the method for screening the disk data in one embodiment;

[0022] Figure 2 A flowchart of the method for screening the disk data in one embodiment;

[0023] Figure 3 A flowchart of the training method of the integrity analysis model in one embodiment;

[0024] Figure 4 A structure diagram of the initial model in one embodiment;

[0025] Figure 5 A structure diagram of the state space in one embodiment;

[0026] Figure 6 A flowchart of the method for screening the disk data based on artificial intelligence in one embodiment;

[0027] Figure 71 is a flow chart of a disk data review method based on artificial intelligence in another embodiment;

[0028] Figure 8 A schematic diagram of the structure of a data collection and preprocessing module in one embodiment;

[0029] Figure 9 is a structural block diagram of a device for screening disk data in one embodiment;

[0030] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] The disk data screening method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, a disk to be tested 102 is in communication with a test terminal 104. A data storage system can store data to be processed by the test terminal 104. The data storage system can be integrated into the test terminal 104.

[0033] Verification terminal 104 obtains disk attribute information of the disk under test 102. The disk attribute information includes disk status information of the disk under test 102 and data attribute information of each of the multiple disk data stored on the disk under test 102. Verification terminal 104 performs data integrity analysis on each disk data based on the disk status information and the data attribute information to determine the expected integrity of each disk data. Verification terminal 104 selects multiple candidate data from each disk data whose expected integrity is less than an integrity threshold. Verification terminal 104 performs a cyclic redundancy check on each candidate data and obtains a check result for the candidate data. Verification terminal 104 identifies target data from each candidate data that fails the check result.

[0034] The disk to be tested 102 may be a memory that stores data using magnetic recording technology.

[0035] The test terminal 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices can include smart watches, smart bracelets, and head-mounted devices. Specifically, the test terminal 104 can communicate with the disk under test 102, and the test terminal 104 can also communicate with the terminal to which the disk under test 102 belongs.

[0036] In one embodiment, as shown in Figure 2 FIG. 1, a method for screening disk data is provided. The method is applied to the inspection terminal 104 in Figure 1 FIG. 1, and includes the following steps.

[0037] S202, obtaining disk attribute information of a disk to be tested.

[0038] The disk can be a storage device that stores data using magnetic recording technology. The disk can store a large amount of binary data and can maintain the data without loss after power failure.

[0039] The disk to be tested can be a disk that needs to be tested for completeness. Specifically, to ensure the safety of the data on the disk, prevent malicious tampering, detect and repair errors, and improve system reliability, the disk to be tested can be checked for completeness. For example, CRC check, hash check, and parity check in RAID technology.

[0040] The disk attribute information can represent information related to the checking dimension of the disk to be tested.

[0041] The disk attribute information can include disk state information of the disk to be tested, and data attribute information of each of a plurality of disk data stored in the disk to be tested.

[0042] The disk state information represents the state information of the disk to be tested itself. For example, the SMART state information of the disk to be tested, and the data damage rate of the disk to be tested.

[0043] The SMART state information of the disk to be tested can represent the self-monitoring, analysis, and reporting technology of the disk. The SMART state includes a series of parameters and information to reflect the health and working status of the hard disk. The main parameters of SMART include at least: attribute ID, attribute name, attribute identifier, manufacturer detection custom content, etc. The attribute ID is used to uniquely identify the SMART attribute. The attribute name can represent the name of a detection item, which is a textual explanation of the ID code. The attribute identifier can be a marker carried by the attribute, such as offline collection, online collection, key data, normal wear attribute, attribute related to performance, etc. The manufacturer detection custom content can represent the current value, worst value, and critical value of the detection item of the disk defined by the disk manufacturer.

[0044] The disk data can represent specific data stored in the disk to be tested. The disk to be tested includes a plurality of disk data. The data attribute information can represent the attribute information of the disk data. For example, the data attribute information can include the data size, write time, previous reading time, reading times, previous review time, user level corresponding to the user of the disk data, etc.

[0045] Specifically, the inspection terminal can acquire disk attribute information of the to-be-tested disk.

[0046] In one of the embodiments, the inspection terminal can directly read the disk attribute information from the to-be-tested disk. The inspection terminal can also indirectly read the disk attribute information from the to-be-tested terminal to which the to-be-tested disk belongs.

[0047] In one of the embodiments, due to the limitation of the computing capability of the inspection terminal, the inspection terminal can read the data attribute information of part of the disk data in the to-be-tested disk. Specifically, the inspection terminal can determine the reading data amount of the disk data according to its own computing capability. The inspection terminal reads the data attribute information of multiple disk data matching the reading data amount in the to-be-tested disk.

[0048] S204, according to the disk state information and the data attribute information, performing data integrity analysis on the disk data to obtain the expected integrity of each disk data.

[0049] The expected integrity can represent the predicted integrity of the disk data. It can be understood that the greater the expected integrity of the disk data, the lower the possibility of data loss of the disk data. Conversely, the smaller the expected integrity of the disk data, the higher the possibility of data loss of the disk data.

[0050] There are various ways to analyze the data integrity of the disk data. Specifically, the inspection terminal can acquire standard disk state information and standard data attribute information. The inspection terminal can compare the state information difference between the disk state information and the standard disk state information, and compare the attribute information difference between each data attribute information and the standard data attribute information. The inspection terminal can determine the expected integrity of each disk data based on the state information difference and the attribute information difference. The greater the state information difference and the attribute information difference, the smaller the expected integrity of the corresponding disk data.

[0051] Specifically, the inspection terminal can also extract features from the disk state information to obtain the disk state features of the to-be-tested disk. The inspection terminal can extract features from the data attribute information to obtain the data attribute features of the disk data. The inspection terminal can input the disk state features and the data attribute features into the integrity analysis model, and determine the predicted integrity of the disk data based on the output of the integrity analysis model.

[0052] S206, in the disk data, screening multiple candidate data whose expected integrity is less than the integrity threshold.

[0053] The completeness threshold value represents a threshold value pre-configured for the expected completeness. The size of the completeness threshold value can be related to the computing capability of the disk to be tested. For example, the greater the computing capability of the disk to be tested, the greater the amount of data that can be processed per unit of time, and thus the greater the expected completeness.

[0054] Specifically, the testing terminal can determine the computing capability of the disk to be tested. The testing terminal can determine the completeness threshold value corresponding to the disk to be tested according to the computing capability. The testing terminal can filter, from the disk data, candidate data whose expected completeness is less than the completeness threshold value.

[0055] In one embodiment, the computing capability of the disk to be tested can include the computing capability of the disk to be tested itself and the computing capability of the terminal to which the disk to be tested belongs. Specifically, the testing terminal determines the computing capability of the disk to be tested itself and the computing capability of the terminal to which the disk to be tested belongs. The testing terminal determines the completeness threshold value corresponding to the disk to be tested according to the two computing capabilities.

[0056] In one embodiment, the computing capability to be tested can also include the computing capability of the testing terminal. Specifically, the testing terminal determines the computing capability of the disk to be tested itself, the computing capability of the terminal to which the disk to be tested belongs, and the computing capability of the testing terminal itself. The testing terminal determines the completeness threshold value corresponding to the disk to be tested according to the three computing capabilities.

[0057] In one embodiment, after obtaining the plurality of computing capabilities, the testing terminal can determine the minimum computing capability from the two (or three) computing capabilities. The testing terminal can determine the completeness threshold value corresponding to the disk to be tested according to the minimum computing capability. The minimum computing capability and the completeness threshold value are positively correlated.

[0058] S208, for each candidate data, performing a cyclic redundancy check on the candidate data to obtain a check result of the candidate data.

[0059] The cyclic redundancy check can represent a fast algorithm for generating a short fixed number of check codes according to network data packets or computer files and the like. The algorithm can be used to detect or check errors that can occur after data transmission or storage, that is, to check the completeness of the disk data.

[0060] The check result can represent the cyclic redundancy check result in the candidate data. The check result can include a cyclic redundancy check failure and a cyclic redundancy check success. The cyclic redundancy check failure represents that the data completeness of the candidate data does not pass, and the candidate data can be reported to facilitate subsequent data maintenance personnel to perform corresponding data recovery operations. The cyclic redundancy check success represents that the data completeness of the candidate data passes the check.

[0061] The cyclic redundancy check includes initializing a CRC check module, calculating a CRC value, storing the CRC value, reading data and checking. Specifically, the verification terminal can configure a CRC module. The verification terminal can calculate a CRC calculation value of the candidate data through a CRC algorithm. The verification terminal can compare the obtained CRC acquisition value of the candidate data with the CRC calculation value. If they are the same, it indicates that the candidate data has not occurred error in the transmission or storage process, and the verification result of the candidate data is passed. Otherwise, it indicates that the data may have been damaged, and the verification result of the candidate data is failed.

[0062] S210, determining target data with a failed verification result from each candidate data.

[0063] The target data indicates the candidate data with a failed verification result. The target data can also indicate data with a low data integrity.

[0064] The target data can be repaired. Specifically, after obtaining the target data, the verification terminal can take measures such as re-writing data or using backup data for repair.

[0065] In the above disk data screening method, the disk attribute information of the to-be-tested disk is obtained, the disk attribute information includes disk state information of the to-be-tested disk and data attribute information of each of the plurality of disk data stored in the to-be-tested disk, the data integrity of each disk data is analyzed according to the disk state information and each data attribute information, and the expected integrity of each disk data is obtained. The expected integrity reflects the estimated integrity of each disk data, which is beneficial to subsequent screening of each data and improves the verification efficiency of the disk data. In the above method, a plurality of candidate data with an expected integrity less than a threshold of integrity is screened from the plurality of disk data, each disk data is preliminarily screened, cyclic redundancy check is performed on each candidate data, the verification result of the candidate data is obtained, and target data with a failed verification result is determined from each candidate data. The data amount of the cyclic redundancy check is reduced, and the verification efficiency is improved.

[0066] In one embodiment, the data integrity of each disk data is analyzed according to the disk state information and each data attribute information, and the expected integrity of each disk data is obtained, including: extracting features from the disk state information to obtain disk state features of the to-be-tested disk, extracting features from the data attribute information to obtain data attribute features of the disk data, inputting the disk state features and the data attribute features into an integrity analysis model, and determining the predicted integrity of the disk data based on the output of the integrity analysis model.

[0067] The disk state features can represent features corresponding to disk state information that can be directly processed by the neural network model. The disk state features can include SMART state features of the to-be-tested disk, data damage features of the to-be-tested disk, and the like.

[0068] The data attribute features can represent features corresponding to data attribute information that can be directly processed by the neural network model. The data attribute features can include data size features, write time features, historical read frequency features, and the like.

[0069] The completeness analysis model can be an analysis model pre-trained for disk data and used to determine the predicted completeness of the disk data. The completeness analysis model can be a pre-trained reinforcement learning model.

[0070] The output of the completeness analysis model can represent a direct output result of the completeness analysis model.

[0071] Specifically, the inspection terminal can determine the predicted completeness of the disk data based on the output of the completeness analysis model.

[0072] In one of the embodiments, the output of the completeness analysis model can include the completeness of the disk data and a confidence corresponding to the completeness. The inspection terminal can obtain a confidence threshold. The inspection terminal can determine the completeness corresponding to the confidence greater than the confidence threshold as the predicted completeness of the disk data.

[0073] In one of the embodiments, in a case where the confidence corresponding to each of the completeness of the to-be-tested disk is less than the confidence threshold, the inspection terminal can determine the completeness corresponding to the maximum confidence among the confidences as the predicted completeness of the disk data.

[0074] There are various ways to perform feature extraction on the disk state information. For example, the inspection terminal can perform feature extraction on the disk state information according to feature extraction parameters in the feature extraction module to obtain the disk state features of the to-be-tested disk. There are various ways to perform feature extraction on the data attribute information. For example, the inspection terminal can perform feature extraction on the data attribute information according to feature extraction parameters in the feature extraction module to obtain the data attribute features of the disk data.

[0075] In one of the embodiments, when performing feature extraction, the inspection terminal can perform preprocessing on the information. For example, the inspection terminal can delete redundant information or merge duplicate information, and the like.

[0076] In this embodiment, the disk state information is feature extracted to obtain the disk state features of the to-be-tested disk, the data attribute information is feature extracted to obtain the data attribute features of the disk data, the disk state features and the data attribute features are input into the completeness analysis model, based on the output of the completeness analysis model, the predicted completeness of the disk data is determined, the feature extraction is performed on the disk state information and the data attribute information, the completeness analysis model is beneficial to predicting the completeness of the disk data, and the accuracy of the predicted completeness can be improved.

[0077] In one embodiment, as shown in the flowchart of the training method of the completeness analysis model, Figure 3 the training process of the completeness analysis model includes:

[0078] S302, obtaining an initial model and a plurality of disk data samples carrying completeness labels.

[0079] As shown in the structural diagram of the initial model, Figure 4 the initial model can be an analysis model to be trained. The initial model can include an environment module and an agent module. The environment module can be an external system for learning and decision-making of the agent module.

[0080] The agent module can be a module constructed by using a DDPG algorithm, and the agent module is used to generate an audit score.

[0081] The agent module includes a policy sub-module and a critic sub-module. The policy sub-module includes a policy network and a target policy network. The policy network receives a current state as input and outputs an action, i.e., a data score. The target policy network receives a next state as input and outputs an action of the next state.

[0082] The critic sub-module is used to calculate a Q value, and the critic sub-module includes a critic network and a target critic network. The critic network receives a current state as input and also receives an action generated by the policy network as additional input, to give a value function under the data state and the action. The target critic network receives a next state as input and also receives a next state action generated by the policy network as additional input.

[0083] In order to simulate the disk environment, as shown in the structural diagram of the state space, Figure 5 a dynamic state space can be set to dynamically update the data state as the disk data is added and the existing data is changed.

[0084] The disk data sample is a to-be-trained sample obtained by using the initial model. The disk data sample includes disk data and a label corresponding to the disk data, i.e., a completeness label.

[0085] The completeness label is the completeness of the disk data in each disk data sample.

[0086] Specifically, the inspection terminal can collect a plurality of initial data samples. The data obtained by the inspection terminal includes damaged data and undamaged data, and determines the integrity label of each initial sample data. The inspection terminal can obtain a disk data sample carrying an integrity label according to the integrity label and each initial sample data. Wherein, the integrity label corresponding to the damaged initial data sample is incomplete and the data integrity is lower than the threshold. The integrity label corresponding to the undamaged initial data sample is complete and the data integrity is higher than the threshold.

[0087] S304, model training is performed on the initial model based on each disk data sample to obtain the predicted integrity of each disk data sample.

[0088] Wherein, the predicted integrity can represent the output result of the initial model for the disk data in the current round of iteration process.

[0089] The process of model training can be a process of updating the parameters in the environment module and the agent module.

[0090] S306, based on the integrity label and the predicted integrity of each disk data sample, the actual integrity sorting and the predicted integrity sorting of each disk data sample are determined.

[0091] Wherein, the actual integrity sorting reflects the sorting result between the integrity labels of each disk data. Specifically, the integrity label of the disk data with higher sorting is greater than the integrity label of the disk data with lower sorting.

[0092] Similarly, the predicted integrity sorting reflects the sorting result between the predicted integrity of each disk data. Specifically, the predicted integrity of the disk data with higher sorting is greater than the predicted integrity of the disk data with lower sorting.

[0093] The way to determine the actual integrity sorting and the predicted integrity sorting of each disk data sample can be descending sorting or ascending sorting, which can be flexibly selected according to the actual situation, which is not limited here.

[0094] S308, the difference between the actual integrity sorting and the predicted integrity sorting is determined, and the step of model training based on each disk data sample on the initial model is returned to perform the next round of iteration.

[0095] Wherein, the difference between the actual integrity sorting and the predicted integrity sorting can represent the overall difference between the two sorting results. The greater the difference can represent the lower the accuracy of the initial model, and vice versa, the smaller the difference can represent the higher the accuracy of the initial model.

[0096] Specifically, the inspection terminal can directly return a step of model training on the initial model based on each disk data sample.

[0097] Specifically, the inspection terminal can also first update each disk data sample to obtain an updated sample. The inspection terminal can return a step of model training on the initial model based on the updated sample.

[0098] S310, in the case of meeting the training end condition, obtaining the completeness analysis model.

[0099] The training end condition can be an end condition configured for the training process of the initial model.

[0100] The training end condition can include that the number of model iterations is greater than a preset value, the difference of the model is less than a preset value, or the change of the difference of the model meets a preset condition. For example, the difference of the Xth model is 0.1, and the difference of the X+1th model is 0.11. At this time, it can be determined that the change of the difference of the Xth model meets the preset condition, and the model after the Xth iteration is determined as the completeness analysis model.

[0101] In this embodiment, the difference between the actual completeness ranking and the predicted completeness ranking is determined, and a step of model training on the initial model based on each disk data sample is returned, the next iteration is performed, and in the case of meeting the training end condition, the completeness analysis model is obtained. The initial model is trained, and the initial model is trained according to the difference between the actual completeness ranking and the predicted completeness ranking. The ranking of each disk data sample in each disk data sample is considered, so that the trained model has the ability to perceive the expected completeness of each disk data, and the analysis accuracy of the completeness analysis model is improved.

[0102] In one embodiment, in the case of meeting the training end condition, obtaining the completeness analysis model includes: obtaining a historical difference between the actual completeness ranking and the predicted completeness ranking in the last iteration round of the current iteration round, obtaining a difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round, and obtaining the completeness analysis model when the variation coefficient of the difference value between the two rounds is less than a preset value within a preset number of rounds, that is, the model is stable.

[0103] The historical difference can represent the difference between the actual completeness ranking and the predicted completeness ranking in the last iteration round.

[0104] In the process of training the model, the difference can be slowly reduced as the number of iterations increases.

[0105] In the case that the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round are less than the preset value in the preset number of rounds, it indicates that the model of the current iteration round is close to or has been fitted, and the model of the current iteration round can be determined as the completeness analysis model.

[0106] Specifically, the inspection terminal can obtain the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round. In the case that the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round are less than the preset value in the preset number of rounds, the inspection terminal can determine the model of the current iteration round as the completeness analysis model. In the case that the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round are greater than the preset value in the preset number of rounds, the inspection terminal can return to the step of training the initial model based on each disk data sample to perform the next iteration.

[0107] In the embodiment, the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round is obtained, and in the case that the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round are less than the preset value in the preset number of rounds, the completeness analysis model is obtained. The numerical relationship between the difference and the historical difference is used to accurately determine whether the model is fitted or close to fitted, and then the completeness analysis model is accurately obtained.

[0108] In one embodiment, the screening method of the disk data further comprises: obtaining the computing capability corresponding to the disk to be tested, determining a completeness threshold value matched with the computing capability, and the completeness threshold value is positively correlated with the computing capability.

[0109] The computing capability corresponding to the disk to be tested can represent the disk read-write capability of the disk to be tested itself, such as IOPS and throughput. The IOPS (Input / Output Operations Per Second) can represent the number of read-write operations that the disk can process in a unit of time. The throughput can represent the amount of data transmitted by the disk per second, usually in units of MB / s (megabytes per second). The throughput reflects the performance of the disk in sequential read-write operations, such as the speed of copying large files or large compressed packages.

[0110] The computing capability of the disk under test can also represent the computing capability of the terminal to which the disk under test belongs, or the computing capability of the test terminal. For example, the number of instructions per second, number of clock cycles per instruction, throughput, CPU frequency, storage capacity, bandwidth, etc. of the terminal to which the disk under test belongs can be measured.

[0111] Higher computing power means more disk data can be screened per unit time, and the corresponding integrity threshold can be higher. While ensuring extraction efficiency, increasing the number of candidate data items further improves screening accuracy and data integrity verification accuracy.

[0112] In this embodiment, the computing power corresponding to the disk to be tested is obtained, and a completeness threshold that matches the computing power is determined. The completeness threshold is positively correlated with the computing power. The completeness threshold and the computing power are matched. While ensuring the extraction efficiency, the number of candidate data items is increased, and the screening accuracy and the accuracy of data integrity verification are further improved.

[0113] In one embodiment, obtaining the computing capacity corresponding to the disk to be tested includes: determining a terminal used to process the disk to be tested, obtaining the disk computing capacity of the disk to be tested and the terminal computing capacity of the terminal, and determining the computing capacity corresponding to the disk to be tested based on the disk computing capacity and the terminal computing capacity.

[0114] The terminal used for processing the disk to be tested may be a terminal to which the disk to be tested belongs, or may be a testing terminal.

[0115] The disk computing capability of the disk to be tested may represent the disk read and write capabilities of the disk to be tested, such as IOPS and throughput.

[0116] Terminal computing capability can refer to the computing capability of the terminal to which the disk under test belongs or the computing capability of the test terminal. For example, terminal computing capability refers to the computing capability of the test terminal. This capability includes the number of instructions per second (I / S), number of clock cycles per instruction (C / I), throughput, CPU frequency, storage capacity, and bandwidth.

[0117] Instructions per second (IPS) indicates the number of instructions a CPU can execute per second. Clock cycles per instruction (CPI) indicates the average number of clock cycles required to execute an instruction. Lower CPI values ​​indicate faster CPU execution. Throughput indicates the amount of information a test terminal can process within a specific time interval and is typically closely related to CPU usage, external interfaces, and I / O. CPU frequency indicates the CPU's main frequency. Storage capacity indicates the amount of storage space within the test terminal. Bandwidth indicates the theoretical peak performance of the system's main memory and is closely related to computing power.

[0118] Specifically, after the terminal obtains the disk computing capability and the terminal computing capability, the terminal can determine the smaller computing capability as the computing capability corresponding to the disk to be tested.

[0119] In one of the embodiments, the testing terminal can determine the computing capability corresponding to the IOPS and the computing capability corresponding to the throughput according to the IOPS and the throughput. The testing terminal determines the smaller computing capability as the disk computing capability.

[0120] In one of the embodiments, the testing terminal can determine the computing capability corresponding to the number of instructions per second, the computing capability corresponding to the number of clock cycles per instruction, the computing capability corresponding to the throughput, the computing capability corresponding to the CPU frequency, the computing capability corresponding to the storage capacity, and the computing capability corresponding to the bandwidth according to the number of instructions per second, the number of clock cycles per instruction, the throughput, the CPU frequency, the storage capacity, and the bandwidth of the testing terminal. The testing terminal determines the smaller computing capability as the terminal computing capability.

[0121] In one of the embodiments, the testing terminal can also determine the computing capability corresponding to the disk to be tested according to the computing capability corresponding to the terminal to which the disk to be tested belongs, the disk computing capability, and the computing capability of the testing terminal.

[0122] In this embodiment, the terminal for processing the disk to be tested is determined, the disk computing capability of the disk to be tested and the terminal computing capability of the terminal are obtained, the computing capability corresponding to the disk to be tested is determined based on the disk computing capability and the terminal computing capability, the accurate computing capability corresponding to the disk to be tested is determined, which is beneficial to determine the accurate completeness threshold, and thus improves the checking accuracy of data completeness.

[0123] In one embodiment, as shown in the method for reviewing disk data based on artificial intelligence, Figure 6 includes the following steps:

[0124] The first part is the model training part, which includes the following steps:

[0125] S602, obtaining an initial model and a plurality of disk data samples carrying completeness labels.

[0126] For example, as shown in the method for reviewing disk data based on artificial intelligence, Figure 7 includes the following steps: step 1, data collection and preprocessing; step 2, establishing a disk data review system model; step 3, training the disk data review system model; step 4, data updating; step 5, continuous learning of the model; and step 6, outputting the sorting result of the data.

[0127] Specifically, step 1: data collection and preprocessing, real-time monitoring of data of a plurality of disks, Figure 8The structure diagram of the data collection and preprocessing module is shown. The data is collected and screened, and the training set and the verification set are set.

[0128] Step 1 includes step 1.1 and step 1.2. Among them, step 1.1, real-time monitoring collects disk data information in a period of time, including: obtaining the SMART state of each disk and the disk data damage rate; Collecting various data of the disk including data size, write time, latest read time, read frequency, last review time, whether the data is deleted, user level and other attribute characteristics as the state information of the disk data.

[0129] Step 1.2 includes preprocessing the obtained information. According to the data damage time, set the damage data score, set the high review score for the data close to damage or already damaged data, and set the low review score for the undamaged data. Use the machine learning model random forest to evaluate the importance of the features for feature selection.

[0130] Step 2: Establish a disk data review system model, such as Figure 4 As shown, it includes an environment module and an agent module. Step 2 includes step 2.1 and step 2.2.

[0131] Among them, step 2.1 environment module is the external system for the agent module to learn and make decisions. It is composed of state space, action space, reward mechanism and termination condition. Step 2.1 includes step 2.1.1 to step 2.1.4.

[0132] Among them, step 2.1.1, the state space of the environment module: the state is the representation of the environment at the current time point, and the state information of the preprocessed data is represented by a vector, where each data feature is an element in the vector. The data in the disk is constantly written and deleted. In order to more accurately simulate the disk environment, a dynamic state space is set here, which dynamically updates the data state as the disk data is added and the existing data changes.

[0133] Step 2.1.2, the action space of the environment module: the action is defined as scoring each data. This means that the policy module needs to learn how to generate a score based on the current data state.

[0134] Step 2.1.3, the reward mechanism of the environment module: is the key part to guide the agent. When the agent's ranking prediction is consistent with the actual result, give positive reward. If the prediction error is large, give negative reward. When the score given by the agent is close to the actual result, give additional positive reward.

[0135] Step 2.1.4, the termination condition of the environment module: reach the predetermined number of iterations, or when the prediction performance of the agent stops improving on the validation set.

[0136] Step 2.2 Agent module, using DDPG algorithm to construct, the agent module includes policy sub-module and evaluation sub-module.

[0137] Step 2.2.1 policy sub-module is used to generate review score. The policy sub-module includes policy network and target policy network. The policy network receives the current state S as input and outputs an action, i.e. disk data score. The target policy network receives the next state S' as input and outputs the action of the next state.

[0138] Step 2.2.2 evaluation sub-module is used to calculate Q value, evaluation sub-module includes evaluation network and target evaluation network. The evaluation network receives the current state S as input, and also receives the action generated by the policy network as additional input, outputs the value function under the data state and action. The target evaluation network receives the next state S' as input, and also receives the next state action generated by the policy network as additional input. Wherein, the Q value can represent the expected reward of the current state.

[0139] Step 3: disk data review system model training: using priority experience replay technology to prioritize the data in experience area, through continuous multiple training of agent, update data review list.

[0140] Step 3.1: initialize the parameters of policy network and evaluation network. Initialize the experience replay buffer to store the experience of agent interacting with the environment.

[0141] Step 3.2: collect state, in the training set, use the current policy network to select an action A according to the current state S at each step. Execute action A, and return the new state S', reward R from the environment interaction. Store the experience (S, A, R, S') into the experience replay buffer, and calculate a priority for each experience according to the TD error. Wherein, the TD error can represent the difference between the reward of the current state and the expected reward of the next state, i.e. the difference between the Q value of the current state and the Q value of the next state.

[0142] Step 3.3: training, from the experience replay buffer, sample a batch of experience data according to the priority. Calculate the Q value of these experience data using the evaluation network, and calculate the target Q value. Calculate the loss of the evaluation network using mean square error. Update the parameters of the evaluation network through back propagation algorithm. Use the Q value gradient of the evaluation network to update the parameters of the policy network to maximize the expected return. Wherein, the sampling can be uniform interval sampling, for example, every two data is sampled in turn.

[0143] Step 3.4: Update target networks, every certain number of training iterations, use the parameters of the current policy network and critic network to update the parameters of the target policy network and target critic network. By soft update method, update the parameters of the current network into the target network by a proportion of 0.001.

[0144] Step 3.5: Iteration, repeat steps 2-4 until the termination condition is reached.

[0145] Step 4: Data update: the increase of data involved in the training set and the change of the existing data state dynamically update the state space of the environment model.

[0146] Step 4.1: Data addition: when new data is added, the state space needs to be expanded accordingly to include the characteristics of the new data.

[0147] Step 4.2: Data update: at fixed time intervals, update the state space of the environment model according to the changes in the data. When data is deleted: set the data to be deleted in the state space. Other data needs to update the SMART state of each disk, the data damage rate of the disk, the data size, the latest reading time, the reading times, the last review time, whether the data is deleted, the user level, and other attribute characteristics.

[0148] Step 5: Model continuous learning, when the state space changes, update the parameters of the policy network and the critic network according to the new state space to find the new optimal policy. This process is iterative until the termination condition is reached.

[0149] Step 5.1: Update state, use the updated state space, at each step use the current policy network to select an action A according to the current state S. Execute action A, and return the new state S' and reward R from the interaction with the environment. Store the experience (S, A, R, S') into the experience replay buffer, and calculate a priority for each experience according to the TD error.

[0150] Step 5.2: Training, sample a batch of experience data from the experience replay buffer according to the priority. Use the critic network to calculate the Q value of these experience data, and calculate the target Q value. Calculate the loss of the critic network using the mean square error. Update the parameters of the critic network through the backpropagation algorithm. Use the Q value gradient of the critic network to update the parameters of the policy network to maximize the expected return.

[0151] Step 5.3: Update target networks, every certain number of iterations, use the parameters of the policy network and critic network to update the parameters of the target policy network and target critic network. By soft update method, update the parameters of the network into the target network by a proportion of 0.001.

[0152] Step 5.4: iteration, repeat step 5.1-step 5.4 until the termination condition is reached.

[0153] Step 6: output data sorting results: verify the trained agent model on the validation set. Finally, the agent will be able to score and sort based on the input data state, and the data sorting is the optimal review method.

[0154] S604, based on each disk data sample, the initial model is trained to obtain the prediction integrity of each disk data sample.

[0155] S606, based on the integrity label and the predicted integrity of each disk data sample, the actual integrity and the predicted integrity of each disk data sample are determined.

[0156] S608, determine the difference between the actual integrity and the predicted integrity.

[0157] S610, obtain the historical difference between the actual integrity and the predicted integrity in the previous round of the current iteration round.

[0158] S612, determine whether the difference between the actual integrity and the predicted integrity in the current iteration round and the historical difference satisfies the variation coefficient less than the preset value within the preset number of rounds, if yes, execute S614, if no, return to S604, and increase the iteration number by one.

[0159] Wherein, the difference between the actual integrity and the predicted integrity in the current iteration round can be represented as the current difference.

[0160] S614, determine the initial model in the current iteration round as the integrity analysis model.

[0161] The second part, the model application part, includes:

[0162] S616, determine the terminal for processing the disk to be tested.

[0163] S618, obtain the disk computing capacity of the disk to be tested, and the terminal computing capacity of the terminal.

[0164] S620, based on the disk computing capacity and the terminal computing capacity, determine the computing capacity corresponding to the disk to be tested.

[0165] S622, determine the integrity threshold matching the computing capacity.

[0166] Wherein, the integrity threshold is positively correlated with the computing capacity.

[0167] S624, obtain the disk attribute information of the disk to be tested.

[0168] The disk attribute information includes disk state information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested.

[0169] S626, feature extraction is performed on the disk state information to obtain disk state features of the disk to be tested.

[0170] S628, feature extraction is performed on the data attribute information to obtain data attribute features of the disk data.

[0171] S630, the disk state features and the data attribute features are input into a completeness analysis model, and based on an output of the completeness analysis model, a predicted completeness of the disk data is determined.

[0172] S632, in each of the disk data, a plurality of candidate data with an expected completeness less than a completeness threshold are screened.

[0173] S634, for each candidate data, a cyclic redundancy check is performed on the candidate data to obtain a check result of the candidate data.

[0174] S636, a target data with a failed check result is determined from each of the candidate data.

[0175] In the embodiment, disk attribute information of the disk to be tested is obtained, the disk attribute information includes disk state information of the disk to be tested and data attribute information of each of a plurality of disk data stored in the disk to be tested, data completeness of each of the disk data is analyzed according to the disk state information and each of the data attribute information to obtain an expected completeness of each of the disk data, the expected completeness reflects an estimated completeness of each of the disk data, which is beneficial to subsequent screening of each of the data and improves the checking efficiency of the disk data. By using the above method, a plurality of candidate data with an expected completeness less than a completeness threshold are screened from each of the disk data, each of the disk data is preliminarily screened, a cyclic redundancy check is performed on each of the candidate data to obtain a check result of each of the candidate data, a target data with a failed check result is determined from each of the candidate data, the amount of data subjected to the cyclic redundancy check is reduced, and the checking efficiency is improved.

[0176] The above method can also bring the following beneficial effects:

[0177] 1. The application assigns reasonable review scores to data and develops the optimal review method. This method not only improves the review efficiency, but also makes reasonable allocation of system resources, making the review work more efficient and energy-saving.

[0178] 2. The application uses artificial intelligence technology to automatically find damaged or potentially damaged data. Compared with traditional methods, it can more accurately identify potential problems, so that timely measures can be taken to avoid data loss or damage.

[0179] 3. The application monitors multiple disk data in a fine-grained manner, and can perform fine-grained monitoring based on multiple characteristics of the data and the SMART state of the disk.

[0180] 4. The application uses continuous learning technology to track the dynamic changes of multiple disk data in real time, and dynamically adjusts the model accordingly, not only quickly adapting to data changes, but also retaining the memory of past learning achievements. This innovation makes the model more accurate and efficient when formulating disk data review optimization methods.

[0181] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0182] Based on the same inventive concept, the embodiments of the present application also provide a disk data screening device for implementing the above-mentioned disk data screening method. The problem-solving implementation scheme provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more disk data screening device embodiments provided below can refer to the limitations of the disk data screening method in the above text, which will not be repeated here.

[0183] In one embodiment, as shown in Figure 9 a disk data screening device is provided, comprising: an acquisition module 902, an analysis module 904, a screening module 906, a verification module 908 and a processing module 910, wherein:

[0184] The acquisition module 902 is configured to acquire disk attribute information of the disk to be tested, wherein the disk attribute information comprises disk state information of the disk to be tested and data attribute information of each of the multiple disk data stored in the disk to be tested;

[0185] The analysis module 904 is configured to perform data integrity analysis on each disk data according to the disk state information and the data attribute information of each disk data, and obtain an expected integrity of each disk data;

[0186] The screening module 906 is configured to screen multiple candidate data with an expected integrity less than a threshold value from the multiple disk data.

[0187] The checking module 908 is configured to perform a cyclic redundancy check on each candidate data to obtain a checking result of the candidate data.

[0188] The processing module 910 is configured to determine, from the candidate data, target data that fails the checking.

[0189] In one of the embodiments, the analysis module 904 is further configured to perform feature extraction on the disk state information to obtain a disk state feature of the disk under test, perform feature extraction on the data attribute information to obtain a data attribute feature of the disk data, and input the disk state feature and the data attribute feature into the completeness analysis model, and determine the predicted completeness of the disk data based on an output of the completeness analysis model.

[0190] In one of the embodiments, the analysis module 904 is further configured to obtain an initial model and a plurality of disk data samples carrying completeness labels, perform model training on the initial model based on the disk data samples to obtain a predicted completeness of each of the disk data samples, determine an actual completeness ranking and a predicted completeness ranking of each of the disk data samples based on the completeness label and the predicted completeness of each of the disk data samples, determine a difference between the actual completeness ranking and the predicted completeness ranking, and return to the step of performing model training on the initial model based on the disk data samples for next iteration, and obtain the completeness analysis model when a training end condition is met.

[0191] In one of the embodiments, the analysis module 904 is further configured to obtain a historical difference between the actual completeness ranking and the predicted completeness ranking in a previous iteration round of a current iteration round, and obtain the completeness analysis model when a difference between the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the historical difference meets a condition that a coefficient of variation is less than a preset value within a preset number of rounds.

[0192] In one of the embodiments, the disk data screening apparatus further includes a threshold determination module configured to obtain a computing capability corresponding to the disk under test, determine a completeness threshold matching the computing capability, and make the completeness threshold positively related to the computing capability.

[0193] In one of the embodiments, the threshold determination module is further configured to determine a terminal used for processing the disk under test, obtain a disk computing capability of the disk under test and a terminal computing capability of the terminal, and determine the computing capability corresponding to the disk under test based on the disk computing capability and the terminal computing capability.

[0194] The modules in the above disk data screening device can be implemented by software, hardware, or a combination thereof, in whole or in part. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0195] In one embodiment, a computer device, which can be a server, has an internal structure as shown in Figure 10 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store disk attribute information data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a disk data screening method.

[0196] Those skilled in the art can understand that Figure 10 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not limit the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0197] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above method steps.

[0198] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the above method steps.

[0199] In one embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the above method steps.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0201] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0202] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for screening disk data, characterized in that: The method comprises: Obtaining disk attribute information of the disk to be tested, wherein the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored on the disk to be tested; Performing data integrity analysis on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data; Screening, from each of the disk data, a plurality of candidate data whose expected completeness is less than a completeness threshold; For each candidate data, performing a cyclic redundancy check on the candidate data to obtain a check result of the candidate data; Determining target data that fails verification results from each candidate data, wherein performing data integrity analysis on each disk data according to the disk status information and each data attribute information to obtain the expected integrity of each disk data includes: Extracting features from the disk status information to obtain disk status features of the disk to be tested; Performing feature extraction on the data attribute information to obtain data attribute features of the disk data; Inputting the disk state features and the data attribute features into an integrity analysis model, and determining the predicted integrity of the disk data based on the output of the integrity analysis model, wherein the training process of the integrity analysis model includes: obtaining an initial model and a plurality of disk data samples carrying integrity labels; The initial model is trained based on each of the disk data samples to obtain the predicted completeness of each of the disk data samples; based on the completeness label and predicted completeness of each of the disk data samples, the actual completeness ranking and the predicted completeness ranking of each of the disk data samples are determined; the difference between the actual completeness ranking and the predicted completeness ranking is determined, and the step of training the initial model based on each of the disk data samples is returned to perform the next round of iteration; when the training end conditions are met, the completeness analysis model is obtained.

2. The method according to claim 1, characterized in that When the training end condition is met, obtaining the integrity analysis model includes: Obtaining a historical difference between the actual completeness ranking and the predicted completeness ranking in a previous round of the current iteration round; When the coefficient of variation of the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the historical difference within a preset round is less than a preset value, a completeness analysis model is obtained.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining the computing capability corresponding to the disk to be tested; Determine a completeness threshold that matches the computing capability; the completeness threshold is positively correlated with the computing capability.

4. The method according to claim 3, characterized in that The obtaining of the computing capability corresponding to the disk to be tested includes: Determining a terminal for processing the disk to be tested; Obtaining the disk computing capability of the disk to be tested and the terminal computing capability of the terminal; The computing capability corresponding to the disk to be tested is determined based on the disk computing capability and the terminal computing capability.

5. A disk data screening device, characterized in that: The device comprises: An acquisition module, configured to acquire disk attribute information of a disk to be tested, wherein the disk attribute information includes disk status information of the disk to be tested and data attribute information of each of a plurality of disk data stored on the disk to be tested; An analysis module, configured to perform data integrity analysis on each disk data according to the disk status information and each data attribute information, to obtain an expected integrity of each disk data; a screening module, configured to screen, from among the disk data, a plurality of candidate data having the expected completeness less than a completeness threshold; A verification module, configured to perform a cyclic redundancy check on each candidate data to obtain a verification result of the candidate data; a processing module, configured to determine target data that fails verification from each of the candidate data, wherein the analysis module is further configured to: performing feature extraction on the disk status information to obtain disk status features of the disk to be tested; performing feature extraction on the data attribute information to obtain data attribute features of the disk data; inputting the disk status features and the data attribute features into an integrity analysis model, and determining the predicted integrity of the disk data based on an output of the integrity analysis model, wherein the analysis module is further configured to: Obtain an initial model and multiple disk data samples with completeness labels; perform model training on the initial model based on each disk data sample to obtain the predicted completeness of each disk data sample; determine the actual completeness ranking and predicted completeness ranking of each disk data sample based on the completeness label and predicted completeness of each disk data sample; determine the difference between the actual completeness ranking and the predicted completeness ranking, and return to the step of training the initial model based on each disk data sample to perform the next round of iteration; when the training end conditions are met, obtain the completeness analysis model.

6. The device according to claim 5, characterized in that The analysis module is also used for: Obtain the historical difference between the actual completeness ranking and the predicted completeness ranking in the previous round of the current iteration round; obtain the completeness analysis model when the difference between the actual completeness ranking and the predicted completeness ranking in the current iteration round and the difference between the historical difference has a coefficient of variation within a preset round that is less than a preset value.

7. The device according to claim 5, characterized in that The disk data screening device further includes a threshold determination module, which is configured to: Obtaining the computing capability corresponding to the disk to be tested; determining a completeness threshold that matches the computing capability; the completeness threshold is positively correlated with the computing capability.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images