Data storage anomaly detection method and system based on cloud platform

Abstract

The invention relates to the technical field of data storage exception analysis, and discloses a data storage exception detection method and system based on a cloud platform, and the method comprises the steps: building an analysis mechanism of cloud platform storage data, carrying out the network transmission monitoring of data transmitted to a storage platform, building a storage end log according to a monitoring result, and storing the storage end log. The preset monitoring index is matched with the storage end log, so that the network transmission abnormity can be warned when the monitoring is abnormal. According to the invention, a perfect cloud platform storage data analysis mechanism is established, the whole data transmission process is monitored carefully, network abnormal signals can be captured in time, and a sufficient basis can be provided for subsequent intelligent matching, so that the real cause of abnormal events can be identified accurately, and abnormal data can be prevented from being stored in the cloud platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage anomaly analysis, in particular to a data storage anomaly detection method and system based on a cloud platform. BACKGROUND

[0002] A data storage cloud platform is a remote data storage, management and processing service system based on advanced cloud computing technology and big data processing capabilities. Through distributed architecture, automatic resource scheduling and multi-level data backup mechanism, it provides a new solution for users to store various structured and unstructured data flexibly and securely without worrying about underlying hardware maintenance. This platform is gradually becoming an important support platform and strategic technology infrastructure for realizing digital transformation and improving data asset value in various industries.

[0003] However, data storage based on a cloud platform usually relies on high-speed network interconnection. When network latency, packet loss or interruption occurs, data synchronization, backup and access processes may be severely affected, resulting in delayed data updates or data consistency problems. Usually, the storage end monitors its own network connection status, response time and abnormal alarms to detect and handle problems in a timely manner. However, if the network problem is at the transmission end, i.e. an exception occurs during data sending, forwarding or transit, the storage end cannot directly detect it because the storage end can only see the received data and delay, etc. indirect indicators, and cannot fully grasp the status of the entire transmission link, resulting in abnormal data in the cloud platform storage. SUMMARY

[0004] In view of the problems existing in the prior art, the purpose of the present application is to provide a data storage anomaly detection method and system based on a cloud platform, which can analyze the storage data of the cloud platform, so as to identify and process abnormal data caused by network abnormal state in the stored data.

[0005] In order to achieve the above purpose, the present application provides the following technical scheme: a data storage anomaly detection method based on a cloud platform, the method comprising the following steps: An analysis mechanism for cloud platform storage data is established to monitor the network transmission of data transmitted to the storage platform, and a storage end log is established according to the monitoring result. By matching the pre-set monitoring indicators with the storage end log, a network transmission anomaly warning can be given when an exception occurs in monitoring; After the data transmission to the storage platform is completed, the time when the transmitted data is written to the storage platform is queried, the time is marked as the total write time, and whether there is a disconnection event in the process of writing the transmitted data to the storage platform is obtained. When no disconnection event occurs, it is marked as normal data and stored; The limit disconnection threshold is set, when the disconnection event occurs, the disconnection index generated by the transmission data written to the storage platform is obtained, and the disconnection index is compared with the limit disconnection threshold, when the disconnection index is less than the limit disconnection threshold, the data is marked as normal data and stored; when the disconnection index is greater than or equal to the limit disconnection threshold, the data is marked as abnormal data; The time stamp of the abnormal data when the disconnection event occurs is obtained, and the hard disk cache change of the storage platform is obtained, whether the abnormal data is caused by cache backlog is judged according to the hard disk cache change, if it is judged that the abnormal data is caused by cache backlog, the abnormal data is changed to normal data and stored; if it is judged that the abnormal data is not caused by cache backlog, the data bit is maintained as abnormal data.

[0006] In some embodiments, the way of determining the disconnection event includes: when the transmission speed is 0kb per second when the transmission data is written to the storage platform, it is recorded as a disconnection event, and the duration of each disconnection event is recorded.

[0007] In some embodiments, the specific way of obtaining the disconnection index is: obtaining the total number of disconnection events Rn generated when the transmission data is written to the storage platform, and summing up the duration of each disconnection event to obtain the total duration of disconnection events Tn, and obtaining the disconnection index Dr=Rn×k1+Tn×k2 through the total number of disconnection events Rn and the total duration of disconnection events Tn, wherein k1 and k2 are the weights of the total number of disconnection events Rn and the total duration of disconnection events Tn respectively.

[0008] In some embodiments, the way of obtaining the hard disk cache change of the storage platform is: obtaining the hard disk cache increment in a preset time period before the disconnection event occurs, the preset time period can be 3%-5% of the total writing time, and the cache increment threshold is set, the hard disk cache increment is compared with the cache increment threshold, and the corresponding response is made according to the comparison result.

[0009] In some embodiments, when the hard disk cache increment is less than or equal to the cache increment threshold, the abnormal data is changed to normal data and stored; when the hard disk cache increment is greater than the cache increment threshold, the data bit is maintained as abnormal data.

[0010] In some embodiments, the way of obtaining the hard disk cache increment includes: obtaining the maximum residual capacity of the hard disk cache in the preset time period, obtaining the maximum cache ratio by dividing the maximum residual capacity by the total capacity of the hard disk cache, obtaining the minimum residual capacity of the hard disk cache in the preset time period, and obtaining the minimum cache ratio by dividing the minimum residual capacity by the total capacity of the hard disk cache, and obtaining the hard disk cache increment by subtracting the maximum cache ratio from the minimum residual capacity.

[0011] In some embodiments, before changing the abnormal data to normal data and storing, whether the change of transmission speed of the abnormal data at the time of the disconnection event is normal should also be obtained, the average transmission speed after the data is transmitted to completion is calculated, and the time when the transmission speed is 0 kb / s is excluded when calculating the average transmission speed, the specific time when the disconnection event occurs is obtained, the transmission speed of the previous 3 seconds is obtained and marked as the adjacent event speed, the transmission speed threshold is set, the transmission speed threshold is 1 / 2 of the average transmission speed, the adjacent event speed is compared with the transmission speed threshold, and a corresponding judgment is made according to the comparison result.

[0012] In some embodiments, if the adjacent event speed is greater than or equal to the transmission speed threshold, the stored data is maintained as abnormal data; and if the adjacent event speed is less than the transmission speed threshold, it is represented that the data transmission has entered a low-speed state before the disconnection event occurs, and the judgment of changing the abnormal data to normal data is maintained.

[0013] The application also provides the following technical solutions: a data storage abnormality detection system based on a cloud platform, comprising: A data analysis module, which comprises an analysis mechanism for establishing cloud platform storage data, network transmission monitoring of data transmitted to the storage platform, and establishment of a storage end log according to the monitoring result, matching of the monitoring index and the storage end log through a preset monitoring index, so as to be able to give a network transmission abnormality warning when the monitoring is abnormal; A disconnection monitoring module, which comprises querying the time when the transmitted data is written to the storage platform after the data transmitted to the storage platform is transmitted to completion, marking the time as the total writing time, and obtaining whether there is a disconnection event in the process of writing the transmitted data to the storage platform, and marking and storing the data as normal data when no disconnection event occurs; A data determination module, which comprises setting a limit disconnection threshold, obtaining a disconnection index generated when the transmitted data is written to the storage platform when a disconnection event occurs, and comparing the disconnection index with the limit disconnection threshold, marking and storing the data as normal data when the disconnection index is less than the limit disconnection threshold, and marking and storing the data as abnormal data when the disconnection index is greater than or equal to the limit disconnection threshold; An abnormality checking module, which comprises obtaining the timestamp of the abnormal data when the disconnection event occurs, and obtaining the hard disk cache change of the storage platform, judging whether the abnormal data is caused by cache accumulation according to the hard disk cache change, changing the abnormal data to normal data and storing if it is judged that the abnormal data is caused by cache accumulation, and maintaining the data as abnormal data if it is judged that the abnormal data is not caused by cache accumulation.

[0014] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the cloud platform-based data storage exception detection method and system.

[0015] Compared with the prior art, the technical solution provided by the application has the following beneficial effects: Firstly, the application establishes a perfect cloud platform storage data analysis mechanism, realizes meticulous monitoring of the whole data transmission process, can timely capture network exception signals, and can provide sufficient basis for subsequent intelligent matching, so as to accurately identify the real reason for the abnormal event and avoid abnormal data being stored in the cloud platform.

[0016] Secondly, the application builds a disconnection index by recording the total number and total duration of disconnection events, and compares the disconnection index with a preset limit disconnection threshold, to evaluate whether the disconnection event is serious, so as to realize real-time judgment of the hard disk cache backlog in the data writing process, and accurately distinguish between the short 0kb / s phenomenon caused by disk I / O waiting and the real network or system failure.

[0017] Thirdly, the application correlates and judges the network transmission state and the hard disk cache state by calculating the transmission speed change of abnormal data before and after the disconnection event, realizes clear distinction between network exception and cache backlog state, and realizes a high-precision and high-efficiency exception monitoring and processing system. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The figure is a logic diagram of the application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0020] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0021] The application provides a cloud platform-based data storage exception detection method, as shown in Figure 1 The method comprises the following steps: Step one, establish the analysis mechanism of cloud platform storage data, monitor the network transmission of data transmitted to the storage platform, the network transmission monitoring includes network delay, packet loss rate, link state, etc., and establish a storage end log according to the monitoring result, match the monitoring index with the storage end log through the preset monitoring index, so as to warn the network transmission exception when the monitoring is abnormal; In the initial stage of cloud platform storage data analysis mechanism, a complete monitoring and analysis system is constructed, which can detect and collect data in real time from transmission, interaction to final storage, so as to capture the subtle network exception in the process of cloud platform storage. This mechanism monitors the key indicators such as network delay, packet loss rate and link state, collects real-time data in the process of network transmission automatically, and makes clear identification of abnormal and normal conditions, so as to form detailed storage end log record in the process of data transmission. The log record includes the timestamp of data transmission, data volume and the interaction state between nodes, and also contains detailed information such as network retransmission, connection recovery and buffer state, so that the real reason of network transmission exception can be identified by intelligent matching with the preset monitoring index in the subsequent process.

[0022] Step two, after the data transmission to the storage platform is completed, query the time when the transmitted data is written to the storage platform, mark this time as the total writing time, and obtain whether there is a disconnection event in the process of writing the transmitted data to the storage platform. When there is no disconnection event, mark the data as normal and store it; The way to determine whether a disconnection event occurs includes: when the transmission speed is 0 kb per second when the transmitted data is written to the storage platform, it is recorded as a disconnection event, and the duration of each disconnection event is recorded: Step three, set a limit disconnection threshold, obtain the disconnection index generated when the transmitted data is written to the storage platform when a disconnection event occurs, and compare the disconnection index with the limit disconnection threshold. When the disconnection index is less than the limit disconnection threshold, mark the data as normal and store it; When the disconnection index is greater than or equal to the limit disconnection threshold, mark the data as abnormal data; Step four, obtain the timestamp of the abnormal data when the disconnection event occurs, and obtain the change of the hard disk cache of the storage platform. According to the change of the hard disk cache, it is judged whether the abnormal data is caused by cache backlog. If it is judged that the abnormal data is caused by cache backlog, the abnormal data is changed to normal data and stored; If it is judged that the abnormal data is not caused by cache backlog, the data is maintained as abnormal data; The specific way to obtain the disconnection index is: obtaining the total number of disconnection events Rn generated when the transmitted data is written to the storage platform, and summing up the time length of each disconnection event to obtain the total time length Tn of the disconnection event, and obtaining the disconnection index Dr = Rn x k1 + Tn x k2 through the total number of disconnection events Rn and the total time length Tn of the disconnection event, wherein k1 and k2 are the weights of the total number of disconnection events Rn and the total time length Tn of the disconnection event respectively.

[0023] The way to obtain the change of the hard disk cache of the storage platform is: obtaining the hard disk cache increment amount in a preset time period before the disconnection event is generated, the preset time period can be 3%-5% of the total writing time, and setting a cache increment amount threshold, comparing the hard disk cache increment amount with the cache increment amount threshold, and making a corresponding response according to the comparison result. Specifically, when the hard disk cache increment amount is less than or equal to the cache increment amount threshold, it means that the cache of the hard disk has a large increment during the writing of the data to the storage platform, which means that the data is accumulated in the cache during the waiting for writing. Then the abnormal data is changed to normal data and stored; when the hard disk cache increment amount is greater than the cache increment amount threshold, it means that the cache of the hard disk does not have a large increment during the writing of the data to the storage platform, which means that the data does not accumulate in the cache during the waiting for writing. Then the data bit is maintained as abnormal data. The reason for the above operation is: the disconnection event is determined by referring to the phenomenon of 0kb / s, and in actual application, the short-term 0kb / s phenomenon may also be caused by the I / O waiting of the disk, and the write operation in the data transmission process is blocked. The application program or system is waiting for the disk to release the write resource. At this time, the new data is mostly temporarily stored in the memory buffer or the operating system cache, and has not been actually written to the disk. Therefore, the write amount displayed in the monitoring tool remains unchanged or increases very slowly, and the monitoring transmission rate is 0kb / s, which indicates that the disk I / O is becoming a bottleneck, and the new write data amount will not be reflected on the disk write total amount in a short period of time. The disconnection event caused by the disk cache accumulation does not affect the abnormality of data transmission and storage. By obtaining the change of the hard disk cache of the storage platform, a more detailed and accurate judgment basis for the actual reason behind the disconnection event is provided, so as to avoid misjudging the short-term 0kb / s phenomenon caused by the disk I / O waiting as a real network or system failure.

[0024] The specific way to obtain the hard disk cache increase amount includes: obtaining the maximum residual capacity of the hard disk cache in a preset time period, using the maximum residual capacity divided by the total capacity of the hard disk cache to obtain the maximum cache ratio, obtaining the minimum residual capacity of the hard disk cache in the preset time period, using the minimum residual capacity divided by the total capacity of the hard disk cache to obtain the minimum cache ratio, and obtaining the difference between the maximum cache ratio and the minimum residual capacity to obtain the hard disk cache increase amount. For example, assuming that the total capacity of the hard disk cache used by the storage cloud platform is 200GB. The preset monitoring time period is set to 5% of the total write time, for example, this period is 10 seconds. In these 10 seconds, the monitoring tool records that the maximum residual capacity of the hard disk cache is 120GB and the minimum residual capacity is 80GB. Then, the maximum cache ratio is calculated, that is, the maximum residual capacity is divided by the total capacity of the hard disk cache, and 120GB / 200GB=60% is obtained. Then, the minimum cache ratio is calculated, that is, 80GB / 200GB=40%. Then, the difference between the two ratios, that is, 60% minus 40%, is obtained, and the hard disk cache increase amount is 20%. Assuming that the pre-set cache increase amount threshold is 25%. In this case, since 20% is less than or equal to 25%, according to the rule: during the 0kb / s disconnection event, the hard disk cache has a large increase (the residual capacity decreases greatly, which means that more data is accumulated in the cache waiting to be actually written to the hard disk), so it can be judged that this disconnection event is likely to be caused by disk I / O waiting, rather than network failure; Therefore, the data marked as abnormal is changed to normal data and continues to be stored. On the contrary, if in another monitoring period, assuming that the recorded maximum residual capacity is 150GB and the minimum residual capacity is 60GB, then the maximum cache ratio is 150GB / 200GB=75%, the minimum cache ratio is 60GB / 200GB=30%, and the difference is 45%, which exceeds the preset threshold of 25%. In this case, it indicates that the decrease of the hard disk cache during the disconnection event is small, and the data is not accumulated in the cache. It may indicate that there is indeed other abnormal problem causing the write delay, so the state of the data as abnormal data should be maintained. Through the above design, the hard disk cache increase amount is obtained and compared with the threshold, which helps to more accurately judge the actual reason of the disconnection event, so as to avoid the short transmission stagnation caused by disk I / O waiting being misjudged as network or system failure.

[0025] Before determining that the abnormal data is caused by cache backlog, the change of transmission speed of the abnormal data at the disconnection event should be obtained, and the abnormal data is changed to normal data and stored. Specifically, the average transmission speed of the data from the transmission completion is calculated, and the time when the transmission speed is 0 kb / s is excluded when calculating the average transmission speed. The specific time when the disconnection event occurs is obtained, the transmission speed of the specific time before 3 seconds is obtained and marked as the adjacent event speed, the transmission speed threshold is set, the transmission speed threshold is half of the average transmission speed, the adjacent event speed is compared with the transmission speed threshold, if the adjacent event speed is greater than or equal to the transmission speed threshold, it represents that the abnormal data still maintains a high and normal transmission speed when entering the disconnection event, which is not the state of slow transmission speed caused by hard disk cache, and it is more likely to be a network problem, and the storage data is maintained as abnormal data. If the adjacent event speed is less than the transmission speed threshold, it represents that the data transmission has entered a low speed state before the disconnection event, which indicates that the transmission speed is not suddenly changed from normal to 0 kb / s, but has been slowly transmitted for a period of time, which is consistent with the phenomenon caused by the increase of data to be written in the hard disk cache, which proves that the abnormal data is not caused by network failure, and the abnormal data is maintained as normal data.

[0026] In general, the present application aims to design a data storage abnormality detection method and system based on a cloud platform. For network problems at the transmission end, it can cause abnormal data storage in the cloud platform. The present application establishes a perfect cloud platform data analysis mechanism to realize detailed monitoring of the whole data transmission process, can timely capture network abnormal signals, and can provide sufficient basis for subsequent intelligent matching, so as to accurately identify the real cause of abnormal events. In the process of data writing into the storage platform, whether the disconnection event with transmission speed of 0 kb / s occurs is judged based on the total writing time, and the disconnection index is constructed by recording the total number and total duration of the disconnection event, and then compared with the preset limit disconnection threshold to evaluate whether the disconnection event is serious. When the disconnection index is lower than the threshold, the data can be regarded as normal data storage even if the disconnection phenomenon occurs, and the high disconnection index identifies abnormal data. By monitoring the change of the remaining capacity of the hard disk cache in the preset time period before the disconnection event, the cache increment is calculated and compared with the set cache increment threshold, so as to realize real-time judgment of the hard disk cache backlog in the data writing process. Its advantage lies in that it can accurately distinguish between the short 0 kb / s phenomenon caused by disk I / O waiting and the real network or system failure. Moreover, by calculating the transmission speed change of the abnormal data before and after the disconnection event, the network transmission state and the hard disk cache state are associated and determined, the network abnormality and cache backlog state are clearly distinguished, and a high-precision and high-efficiency abnormality monitoring and processing system is realized.

[0027] The application further provides a cloud platform-based data storage abnormality detection system, comprising: a data analysis module, which comprises an analysis mechanism for establishing cloud platform storage data, network transmission monitoring for data transmitted to the storage platform, and storage end log establishment according to the monitoring result, matching of the monitoring index and the storage end log through a preset monitoring index, so as to give a network transmission abnormality warning when an abnormality occurs in the monitoring; a disconnection monitoring module, which comprises querying the time when the transmitted data is written into the storage platform after the data transmission to the storage platform is completed, marking the time as the total writing time, and acquiring whether there is a disconnection event in the process of writing the transmitted data into the storage platform, marking the data as normal data and storing the data when no disconnection event occurs; a data determination module, which comprises setting a limit disconnection threshold, acquiring the disconnection index generated when the transmitted data is written into the storage platform when a disconnection event occurs, and comparing the disconnection index with the limit disconnection threshold, marking the data as normal data and storing the data when the disconnection index is less than the limit disconnection threshold, and marking the data as abnormal data when the disconnection index is greater than or equal to the limit disconnection threshold; an abnormality checking module, which comprises acquiring the timestamp when the abnormal data generates the disconnection event, and acquiring the hard disk cache change of the storage platform, judging whether the abnormal data is caused by cache backlog according to the hard disk cache change, changing the abnormal data to normal data and storing the data if it is judged that the abnormal data is caused by cache backlog, and maintaining the data as abnormal data if it is judged that the abnormal data is not caused by cache backlog.

[0028] The processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the present disclosure. Embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but not limited to, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire, optical cable, RF or any suitable combination of the above.

[0029] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0030] Those skilled in the art should understand that the above description is only a specific embodiment of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for detecting data storage anomalies based on a cloud platform, characterized in that, The method includes the following steps: Establish an analysis mechanism for cloud platform stored data, monitor the network transmission of data transmitted to the storage platform, and establish storage logs based on the monitoring results. Match the storage logs with preset monitoring indicators to issue warnings for network transmission anomalies when monitoring results are abnormal. After the data transmission to the storage platform is completed, query the time when the transmitted data was written to the storage platform, mark this time as the total writing time, and obtain whether there is a disconnection event during the process of transmitting data to the storage platform. If no disconnection event occurs, mark it as normal data and store it. Set a limit disconnection threshold. When a disconnection event occurs, obtain the disconnection index generated by the transmission of data written to the storage platform, and compare the disconnection index with the limit disconnection threshold. When the disconnection index is less than the limit disconnection threshold, the data is marked as normal data and stored; when the disconnection index is greater than or equal to the limit disconnection threshold, the data is marked as abnormal data. Obtain the timestamp of the disconnection event that caused the abnormal data, and obtain the changes in the hard disk cache of the storage platform. Based on the changes in the hard disk cache, determine whether the abnormal data was caused by cache backlog. If it is determined that the abnormal data was caused by cache backlog, change the abnormal data to normal data and store it. If it is determined that the abnormal data is not caused by cache backlog, then the judgment that the data bit is abnormal data is maintained.

2. The data storage anomaly detection method based on a cloud platform according to claim 1, characterized in that, The methods for determining when a disconnection event occurs include: when the transmission speed is 0kb per second while data is being written to the storage platform, it is recorded as a disconnection event, and the duration of each disconnection event is also recorded.

3. The data storage anomaly detection method based on a cloud platform according to claim 2, characterized in that, The specific method to obtain the disconnection index is as follows: obtain the total number of disconnection events Rn generated when the transmitted data is written to the storage platform, and sum the duration of each disconnection event to obtain the total duration of the disconnection event Tn. The disconnection index Dr = Rn×k1 + Tn×k2 is obtained through the total number of disconnection events Rn and the total duration of the disconnection event Tn, where k1 and k2 are the weights of the total number of disconnection events Rn and the total duration of the disconnection event Tn, respectively.

4. The data storage anomaly detection method based on a cloud platform according to claim 3, characterized in that, The method for obtaining changes in the hard disk cache of the storage platform is as follows: obtain the increase in hard disk cache within a preset time period before the disconnection event occurs. The preset time period can be 3%-5% of the total write time. Set a cache increase threshold, compare the hard disk cache increase with the cache increase threshold, and take appropriate action based on the comparison result.

5. The data storage anomaly detection method based on a cloud platform according to claim 4, characterized in that, When the increase in hard disk cache is less than or equal to the cache increase threshold, the abnormal data is changed to normal data and stored; when the increase in hard disk cache is greater than the cache increase threshold, the data is maintained as abnormal data.

6. The data storage anomaly detection method based on a cloud platform according to claim 5, characterized in that, The methods for obtaining the increase in hard disk cache include: obtaining the maximum remaining capacity of hard disk cache within a preset time period, dividing the maximum remaining capacity by the total capacity of hard disk cache to obtain the maximum cache percentage, obtaining the minimum remaining capacity of hard disk cache within a preset time period, dividing the minimum remaining capacity by the total capacity of hard disk cache to obtain the minimum cache percentage, and calculating the difference between the maximum cache percentage and the minimum remaining capacity to obtain the increase in hard disk cache.

7. The data storage anomaly detection method based on a cloud platform according to claim 6, characterized in that, Before determining that the abnormal data is caused by cache backlog, changing the abnormal data to normal data, and storing it, it is also necessary to determine whether the change in transmission speed of the abnormal data during the disconnection event is normal, calculate the average transmission speed from the time the data is transmitted to the completion of the transmission, and exclude the time when the transmission speed is 0kb / s when calculating the average transmission speed, obtain the specific time when the abnormal data causes the disconnection event, obtain the transmission speed of 3 seconds before the specific time, and mark it as the near event speed, set a transmission speed threshold, the transmission speed threshold is taken as 1 / 2 of the average transmission speed, compare the near event speed with the transmission speed threshold, and make corresponding judgments based on the comparison results.

8. The data storage anomaly detection method based on a cloud platform according to claim 7, characterized in that, If the speed of the nearest event is greater than or equal to the transmission speed threshold, the stored data is kept as abnormal data; if the speed of the nearest event is less than the transmission speed threshold, it means that the data transmission has already entered a low-speed state before the disconnection event occurs, and the judgment of changing abnormal data to normal data is maintained.

9. A data storage anomaly detection system based on a cloud platform, characterized in that, A cloud platform-based data storage anomaly detection method according to any one of claims 1-8, comprising: The data analysis module includes establishing an analysis mechanism for cloud platform stored data, monitoring network transmission of data transmitted to the storage platform, and establishing storage logs based on monitoring results. By matching the storage logs with preset monitoring indicators, warnings can be issued for network transmission anomalies when monitoring results are abnormal. The disconnection monitoring module includes querying the time when the transmitted data is written to the storage platform after the data transmission to the storage platform is completed, marking the time as the total writing time, and obtaining whether there is a disconnection event during the process of transmitting data to the storage platform. If no disconnection event occurs, it is marked as normal data and stored. The data determination module includes setting a limit disconnection threshold. When a disconnection event occurs, it obtains the disconnection index generated by the transmission of data written to the storage platform and compares the disconnection index with the limit disconnection threshold. When the disconnection index is less than the limit disconnection threshold, the data is marked as normal data and stored. When the disconnection index is greater than or equal to the limit disconnection threshold, the data is marked as abnormal data. The anomaly detection module includes obtaining the timestamp of the disconnection event when the abnormal data occurs, obtaining the changes in the hard disk cache of the storage platform, and determining whether the abnormal data is caused by cache backlog based on the changes in the hard disk cache. If the abnormal data is determined to be caused by cache backlog, the abnormal data is changed to normal data and stored; if the abnormal data is determined not to be caused by cache backlog, the judgment that the data is abnormal is maintained.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor to implement the cloud platform-based data storage anomaly detection method according to any one of claims 1-8.

Images