A data processing method and system for sharing diversified data sources of documents
By performing feature partitioning, abnormal detection and thermal analysis on multi-source heterogeneous data, disaster recovery segmented storage sequence is determined, and data disaster recovery under extreme network conditions is achieved, solving the problem of data loss caused by excessive load on the server storage node. , improves the stability of the storage server.
Patent Information
- Application Number
- CN202410753978.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-12
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-06-12
Smart Images

Figure CN118503221B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and more specifically, to a data processing method and system for sharing diversified data sources of documents. Background Art
[0002] Data processing covers a variety of technologies and methods for collecting, processing, analyzing and storing data, including data cleaning, conversion, integration, mining and visualization. The goal of data processing technology is to extract useful information from massive data for decision making, predictive analysis and business optimization. The main technologies include ETL (extraction, transformation, loading), data warehouse, data lake, big data processing (such as Hadoop, Spark), machine learning and artificial intelligence algorithms. In recent years, the development of cloud computing, edge computing and the Internet of Things has made data processing technology more diverse and efficient. Data processing technology has been applied in all walks of life, including finance, medical care, retail, manufacturing and other fields, and has become a key tool for organizations to improve efficiency, reduce costs and innovate.
[0003] Data processing of diversified data sources for shared documents involves obtaining data from various sources, such as text, images, videos, etc., using machine learning technology to mine the correlation patterns between data after data cleaning, and then selecting appropriate storage methods (such as databases, cloud storage, etc.) to store different data in a hierarchical manner. However, in the prior art, due to extreme network conditions (such as high network latency, increased packet loss rate or limited bandwidth), a large number of users read or write data on the same storage node on the server in a short period of time, which may cause some storage nodes of the server to be overloaded, resulting in data loss or system crashes caused by the storage nodes. Therefore, how to perform data disaster recovery for the data of the server storage nodes under extreme network conditions, thereby improving the stability of the server storing diversified data, has become a difficult problem faced by the industry. Summary of the invention
[0004] The present application provides a data processing method and system for a shared document diversified data source, which can realize data disaster recovery for data of server storage nodes under extreme network conditions, thereby improving the stability of the server storing diversified data.
[0005] In a first aspect, the present application provides a data processing method for a shared document diversified data source, comprising the following steps:
[0006] Acquire data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data;
[0007] Partitioning the multi-source heterogeneous data according to user access characteristics to different types of data to obtain cold data areas and hot data areas;
[0008] Performing anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determining the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determining the multi-source heterogeneous gain entropy of the target shared document through a preset data gain value and the heterogeneous feature distribution domain;
[0009] Extracting the access heat records of users accessing the target shared document from the hot data area, and then determining the data disaster recovery coefficient of each user under extreme network conditions according to the access heat records, and determining the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all the data disaster recovery coefficients;
[0010] The data of the diversified data sources of the target shared document are hierarchically stored according to the disaster recovery segment storage sequence.
[0011] In some embodiments, partitioning the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas specifically includes:
[0012] Obtain user access characteristics to different types of data;
[0013] Mapping the multi-source heterogeneous data through the access feature to obtain multi-source heterogeneous mapping data;
[0014] Based on a pre-trained hot and cold data classification model, the access features are marked as hot and cold data to obtain hot and cold data access features;
[0015] The multi-source heterogeneous mapping data is partitioned into cold and hot data according to the cold and hot data access characteristics to obtain a cold data area and a hot data area.
[0016] In some embodiments, performing anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data specifically includes:
[0017] Obtain user access characteristics to different types of data;
[0018] Extracting access time series of different types of data by users in the cold data area through the access features;
[0019] Determining a plurality of abnormal data marking points in the cold data area according to the access time series;
[0020] Determine the abnormal data critical value based on all abnormal data marking points;
[0021] The heterogeneous abnormal data in the cold data area is determined by the abnormal data critical value.
[0022] In some embodiments, determining the multi-source heterogeneous gain entropy of the target shared document by using the preset data gain value and the heterogeneous feature distribution domain specifically includes:
[0023] Performing feature extraction on all heterogeneous feature distribution vectors in the heterogeneous feature distribution domain to obtain a plurality of heterogeneous feature factors;
[0024] The multi-source heterogeneous gain entropy of the target shared document is determined according to all heterogeneous feature factors and preset data gain values.
[0025] In some embodiments, determining the data disaster recovery coefficient of each user under extreme network conditions according to the access popularity record specifically includes:
[0026] Determining an extreme network threshold of each user under extreme network conditions according to the access heat record;
[0027] Obtaining a dissimilarity set of the target shared document;
[0028] Data disaster recovery is performed on all extreme network thresholds through the dissimilarity set to obtain a data disaster recovery coefficient for each user under extreme network conditions.
[0029] In some embodiments, determining the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients specifically includes:
[0030] Perform linear fitting on all data disaster recovery coefficients to obtain a data disaster recovery curve;
[0031] Determine the disaster recovery segment amount of each user corresponding to each data disaster recovery coefficient according to the data disaster recovery curve;
[0032] The disaster recovery segment storage sequence of the target shared document is determined according to the multi-source heterogeneous gain entropy and all disaster recovery segment quantities.
[0033] In some embodiments, performing hierarchical storage processing on the data of the target shared document diversified data sources according to the disaster recovery segment storage sequence specifically includes:
[0034] Determine hierarchical storage coefficients for different types of data from a target shared document diversified data source according to the disaster recovery segment storage sequence;
[0035] The data corresponding to the diversified data sources of the target shared document are stored and processed through all hierarchical storage coefficients.
[0036] In a second aspect, the present application provides a data processing system for sharing diversified data sources of documents, including:
[0037] The acquisition module is used to acquire data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data;
[0038] A processing module, used for partitioning the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas;
[0039] The processing module is further used to perform anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determine the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determine the multi-source heterogeneous gain entropy of the target shared document through the preset data gain value and the heterogeneous feature distribution domain;
[0040] The processing module is further used to extract the access heat records of users accessing the target shared document from the hot data area, and then determine the data disaster recovery coefficient of each user under extreme network conditions according to the access heat records, and determine the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients;
[0041] The execution module is used to perform hierarchical storage processing on the data of the diversified data sources of the target shared document according to the disaster recovery segment storage sequence.
[0042] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned data processing method for a shared document diversified data source.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method for the shared document diversified data source is implemented.
[0044] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:
[0045] In the data processing method and system for the diversified data sources of a shared document provided by the present application, data from different data sources are obtained from a target shared document to obtain multi-source heterogeneous data; the multi-source heterogeneous data are partitioned according to the user's access characteristics to different types of data to obtain a cold data area and a hot data area; anomaly detection is performed on different types of data in the cold data area to obtain heterogeneous anomaly data, and then the heterogeneous feature distribution domain of the user in the process of accessing the target shared document is determined according to the heterogeneous anomaly data, and the multi-source heterogeneous gain entropy of the target shared document is determined by a preset data gain value and the heterogeneous feature distribution domain; the access heat record of the user accessing the target shared document is extracted from the hot data area, and then the data disaster recovery coefficient of each user under extreme network conditions is determined according to the access heat record, and the disaster recovery segment storage sequence of the target shared document is determined based on the multi-source heterogeneous gain entropy and all the data disaster recovery coefficients; and the data of the diversified data sources of the target shared document are hierarchically stored according to the disaster recovery segment storage sequence.
[0046] It can be seen that in the present application, the disaster recovery segmented storage sequence of the target shared document can be determined based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients; wherein, first, a heterogeneous feature distribution domain for describing the abnormal behavior characteristics of different users in the process of accessing the target shared document is extracted from the heterogeneous abnormal data, and then a multi-source heterogeneous gain entropy for reducing the uncertainty degree of the abnormal information of the target shared document is extracted from the heterogeneous feature distribution domain through a preset data gain value; secondly, the access heat record is quantified to obtain a data disaster recovery coefficient that describes the recovery capability of each user when the data is lost under extreme network conditions; finally, the data of the diversified data sources of the target shared document are hierarchically stored according to the disaster recovery segmented storage sequence; in summary, the scheme of the present application can realize data disaster recovery for the data of the server storage node under extreme network conditions, thereby improving the stability of the server storing diversified data. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is an exemplary flow chart of a data processing method for a shared document diversified data source according to some embodiments of the present application;
[0048] Figure 2 is a schematic diagram of a process for determining a cold data area and a hot data area in some embodiments of the present application;
[0049] Figure 3 is a schematic diagram of a process for determining a data disaster recovery coefficient in some embodiments of the present application;
[0050] Figure 4 is a structural block diagram of a data processing system for sharing diversified data sources of documents in some embodiments of the present application;
[0051] Figure 5 It is a structural diagram of a computer device for implementing a data processing method for a shared document with diversified data sources as shown in some embodiments of the present application. DETAILED DESCRIPTION
[0052] The core of this application is to obtain data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data; partition the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas; perform anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determine the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determine the multi-source heterogeneous gain entropy of the target shared document through the preset data gain value and the heterogeneous feature distribution domain; extract the access heat record of the user accessing the target shared document from the hot data area, and then determine the data disaster recovery coefficient of each user under extreme network conditions based on the access heat record, and determine the disaster recovery segmented storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients; perform hierarchical storage processing on the data of the diversified data sources of the target shared document according to the disaster recovery segmented storage sequence. The scheme of this application can realize data disaster recovery for the data of the server storage node under extreme network conditions, thereby improving the stability of the server storing diversified data.
[0053] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a data processing method for a shared document diversified data source according to some embodiments of the present application. The data processing method 100 for a shared document diversified data source mainly includes the following steps:
[0054] In step 101, data from different data sources are acquired from a target shared document, thereby obtaining multi-source heterogeneous data.
[0055] In specific implementation, data from different data sources are obtained from the network server of the target shared document, and the obtained data is used as multi-source heterogeneous data. It should be noted that the multi-source heterogeneous data in the present application is composed of text data, digital data, table data, image data and link data. The text data is composed of multiple text components, wherein the text components in the text data include project description, task description, project goal statement, project report summary and project review feedback data; the digital data is composed of multiple digital components, wherein the digital components in the digital data include project quantity, project amount and project date data; the table data is composed of multiple table components, wherein the table components in the table data include project schedule, project resource allocation table and project list table; the image data is composed of multiple image components, wherein the image components in the image data include project flow chart and project organization chart; the link data is composed of multiple link components, wherein the link components in the link data include web page links, file links and database query links.
[0056] In addition, it should be noted that in the present application, one user corresponds to multiple text components, multiple numeric components, multiple table components, multiple image components and multiple link components, and the set consisting of multiple text components, multiple numeric components, multiple table components, multiple image components and multiple link components is regarded as a multi-source heterogeneous group, so one user corresponds to one multi-source heterogeneous group.
[0057] In step 102, the multi-source heterogeneous data is partitioned according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas.
[0058] In some embodiments, reference Figure 2 As shown, the figure is a schematic diagram of the process of determining the cold data area and the hot data area in some embodiments of the present application. In this embodiment, the multi-source heterogeneous data is partitioned according to the user's access characteristics to different types of data, and the cold data area and the hot data area can be obtained by the following steps:
[0059] First, in step 1021, the user's access characteristics to different types of data are obtained;
[0060] Next, in step 1022, the multi-source heterogeneous data is mapped by using the access feature to obtain multi-source heterogeneous mapped data;
[0061] Then, in step 1023, the access features are labeled as cold and hot data based on the pre-trained cold and hot data classification model to obtain cold and hot data access features;
[0062] Finally, in step 1024, the multi-source heterogeneous mapping data is partitioned into cold and hot data according to the cold and hot data access characteristics to obtain a cold data area and a hot data area.
[0063] In specific implementation, the user's access characteristics to different types of data are obtained from the access log in the network server of the target shared document. It should be noted that the access characteristics described in the present application are composed of access frequency data, access time data and visitor identity data of different types of data, wherein the access frequency data is composed of multiple access frequencies, the access time data is composed of multiple access times, wherein the access time is in the form of a timestamp, and the visitor identity data is composed of multiple visitor IDs.
[0064] In addition, it should be noted that in this application, one user corresponds to multiple access ratings, multiple access times, and multiple visitor IDs, and each visitor ID is the same, and a set of multiple access ratings, multiple access times, and multiple visitor IDs is taken as an access feature group. Therefore, one user corresponds to one access feature group.
[0065] In specific implementation, the multi-source heterogeneous data is mapped through the access feature to obtain the multi-source heterogeneous mapping data, which can be achieved in the following manner, namely: the access frequency data, access time data and visitor identity data in the access feature are mapped one-to-one with the text data, digital data, table data, image data and link data in the multi-source heterogeneous mapping data, so as to obtain the multi-source heterogeneous mapping data, for example: the visitor IDs of all access feature groups in the access feature are associated with the user IDs corresponding to the multi-source heterogeneous groups.
[0066] In specific implementation, the access features are marked as hot and cold data based on the pre-trained hot and cold data classification model, and the hot and cold data access features can be obtained in the following manner, namely: first, historical access features are obtained, and the historical access features are marked according to the hot and cold data classification standards to obtain historical access feature marked data, wherein the hot and cold data classification standard is that if the data is accessed more than 20 times per minute, it is marked as hot data, otherwise it is marked as cold data; secondly, the historical access feature marked data is used as input data, and a hot and cold data classification model is obtained after training with a logistic regression (LR) model; then, the hot and cold data classification model is used to perform a forward operation on the access features, and the vector obtained after the operation is used as the hot and cold data access features. Other methods may also be used in other embodiments, which are not limited here.
[0067] It should be noted that the cold and hot data access features described in the present application represent the vector obtained after using the historical access features to operate through the cold and hot data classification model, and each value in the vector is used as the cold and hot data access feature value. The cold and hot data access feature values are used to measure the cold and hot properties of the data in the multi-source heterogeneous mapping data corresponding to the user, and the cold and hot properties are only of two types: cold or hot.
[0068] In a specific implementation, the multi-source heterogeneous mapping data is partitioned into cold and hot data according to the cold and hot data access characteristics, and the cold data area and the hot data area can be obtained by the following method, namely: a cold and hot data access characteristic value is selected from the cold and hot data access characteristics, and if the cold and hot property of the cold and hot data access characteristic value is cold, then the multi-source heterogeneous group of the user corresponding to the cold and hot data access characteristic value in the multi-source heterogeneous mapping data is marked as cold, and the multi-source heterogeneous group is used as the multi-source heterogeneous cold data group, and if the cold and hot data access characteristic value is If the cold and hot property of the characteristic value is hot, the multi-source heterogeneous group of the user corresponding to the cold and hot data access characteristic value in the multi-source heterogeneous mapping data is marked as hot, and the multi-source heterogeneous group is used as a multi-source heterogeneous hot data group. Repeat the above steps to obtain multiple multi-source heterogeneous cold data groups and multiple multi-source heterogeneous hot data groups, and take the set composed of all multi-source heterogeneous cold data groups as the cold data area, and take the set composed of all multi-source heterogeneous hot data groups as the hot data area. In other embodiments, other methods can also be used to achieve this, which will not be repeated here.
[0069] It should be noted that the cold data area described in this application refers to a collection area composed of data that is not frequently accessed or needs to be stored for a long time but does not need to be accessed frequently, and the hot data area refers to a collection area composed of data that is frequently accessed or requires a quick response. By dividing the cold data area and the hot data area, the use of storage resources can be optimized and higher value data can be accessed faster, thereby improving the performance and efficiency of the overall storage system.
[0070] In step 103, anomaly detection is performed on different types of data in the cold data area to obtain heterogeneous anomaly data, and then the heterogeneous feature distribution domain of the user in the process of accessing the target shared document is determined based on the heterogeneous anomaly data, and the multi-source heterogeneous gain entropy of the target shared document is determined by the preset data gain value and the heterogeneous feature distribution domain.
[0071] In some embodiments, performing anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data may be achieved by using the following steps:
[0072] Obtain user access characteristics to different types of data;
[0073] Extracting access time series of different types of data by users in the cold data area through the access features;
[0074] Determining a plurality of abnormal data marking points in the cold data area according to the access time series;
[0075] Determine the abnormal data critical value based on all abnormal data marking points;
[0076] The heterogeneous abnormal data in the cold data area is determined by the abnormal data critical value.
[0077] In specific implementation, the following method can be used to extract the access time series of users to different types of data in the cold data area through the access features, namely: the access time of users to different types of data in the cold data area is selected from the access features, and all the selected access times are sorted in chronological order, and the sorted sequence is used as the access time series of users to different types of data in the cold data area; the following method can be used to determine multiple abnormal data points in the cold data area based on the access time series, namely: the access time corresponding to the median of the access time series can be used as the central mark point, and the k access times closest to the central mark point are all used as abnormal data mark points in the cold data area. It should be noted that the value of k in this application can be set according to actual needs. The larger the value of k, the more heterogeneous abnormal data there are, the more storage resources need to be used, but the more accurate the division of heterogeneous abnormal data is.
[0078] In specific implementation, the abnormal data critical value represents the threshold value for dividing abnormal data, which can determine whether a data point belongs to abnormal data. As a preferred embodiment, determining the abnormal data critical value based on all abnormal data marking points can be implemented in the following manner, namely: selecting the abnormal data marking point farthest from the central marking point from all abnormal data marking points, and subtracting the value of the selected abnormal data marking point from the value of the central marking point, and taking twice the value obtained by the difference as the abnormal data critical value. Other methods can also be used in other embodiments, which are not limited here.
[0079] In specific implementation, determining the heterogeneous abnormal data in the cold data area by the abnormal data critical value can be implemented in the following manner, namely: first, selecting an access time from the access time sequence, if the difference between the value of the access time and the value of the central mark point is greater than the abnormal data critical value, then taking the multi-source heterogeneous cold data group of the user corresponding to the access time in the cold data area as the heterogeneous abnormal data group, then repeating the above steps to obtain the multi-source heterogeneous cold data groups of the users corresponding to the remaining access times in the cold data area as the heterogeneous abnormal data group, finally, taking the set consisting of all heterogeneous abnormal data groups as the heterogeneous abnormal data in the cold data area, other methods can also be used in other embodiments, which will not be repeated here.
[0080] In some embodiments, determining the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous abnormal data can be implemented by the following steps:
[0081] Determine a dissimilarity set of a target shared document according to the heterogeneous abnormal data;
[0082] Clustering all dissimilarities in the dissimilarity set to obtain multiple dissimilarity clustering centers;
[0083] Determine the cluster feature distribution sequence through all dissimilarity cluster centers;
[0084] The heterogeneous feature distribution domain of the user in the process of accessing the target shared document is determined according to the cluster feature distribution sequence.
[0085] In specific implementation, determining the dissimilarity set of the target shared document based on the heterogeneous exception data can be implemented in the following manner, namely: first, a heterogeneous exception data group is selected from the heterogeneous exception data, and the Jaccard similarity can be used to calculate the similarity of all remaining heterogeneous exception data groups in the heterogeneous exception data. Secondly, the absolute value is taken after subtracting the maximum similarity from 1, and the value obtained after taking the absolute value is used as the dissimilarity of the heterogeneous exception data group. The above steps are repeated to obtain the dissimilarity of the remaining heterogeneous exception data groups. Then, a set consisting of all the obtained dissimilarity is used as the dissimilarity set of the target shared document. Other methods can also be used in other embodiments, which are not limited here.
[0086] It should be noted that the dissimilarity in the dissimilarity set described in the present application represents the degree of dissimilarity between different heterogeneous abnormal data in the heterogeneous abnormal data group. The greater the dissimilarity, the greater the degree of dissimilarity between different heterogeneous abnormal data in the heterogeneous abnormal data group, and the smaller the dissimilarity, the smaller the degree of dissimilarity between different heterogeneous abnormal data in the heterogeneous abnormal data group.
[0087] In addition, it should be noted that in the present application, a common clustering algorithm can be used to cluster all the dissimilarities in the dissimilarity set to obtain multiple dissimilarity clustering centers, where common clustering algorithms include K-means clustering algorithm, hierarchical clustering algorithm and DBSCAN clustering algorithm, etc., which are not limited here. In addition, each dissimilarity clustering center corresponds to a dissimilarity, and other methods can also be used in other embodiments to achieve this, which will not be repeated here.
[0088] In specific implementation, the clustering feature distribution sequence determined by all the dissimilarity cluster centers can be implemented in the following manner, namely: first, a dissimilarity cluster center is selected, and secondly, the Euclidean distance between the dissimilarity cluster center and the remaining dissimilarity cluster centers is calculated, and the maximum value of the Euclidean distance between the dissimilarity cluster center and the remaining dissimilarity cluster centers is used as the clustering feature distribution value of the dissimilarity cluster center, and then, the above steps are repeated to obtain the clustering feature distribution values of the remaining dissimilarity cluster centers, and finally, all the clustering feature distribution values are sorted in ascending order, and the sequence obtained after sorting is used as the clustering feature distribution sequence. Other methods can also be used in other embodiments, which are not limited here.
[0089] It should be noted that the clustering feature distribution value in the clustering feature distribution sequence described in the present application represents a parameter value for measuring the degree of separation of the dissimilarity cluster centers. The smaller the clustering feature distribution value, the smaller the degree of separation of the dissimilarity cluster centers and the better the clustering effect. The larger the clustering feature distribution value, the greater the degree of separation of the dissimilarity cluster centers and the worse the clustering effect.
[0090] In specific implementation, determining the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the cluster feature distribution sequence can be implemented in the following manner, namely: first, selecting the first cluster feature distribution value from the cluster feature distribution sequence, subtracting the first cluster feature distribution value from all the remaining cluster feature distribution values in the arrangement order of the remaining cluster feature distribution values and taking the absolute value, and forming a vector with all the values obtained after taking the absolute value in the order of subtraction and taking the absolute value; secondly, using the obtained vector as the heterogeneous feature distribution vector of the cluster feature distribution value, and using the values in the heterogeneous feature distribution vector as the heterogeneous feature distribution values; then, repeating the above steps to obtain the heterogeneous feature distribution vector of the second cluster feature distribution value in turn, until the heterogeneous feature distribution vector of the last cluster feature distribution value is obtained; finally, using the set composed of all the heterogeneous feature distribution vectors as the heterogeneous feature distribution domain of the user in the process of accessing the target shared document; other methods can also be used in other embodiments, which will not be repeated here.
[0091] It should be noted that the heterogeneous feature distribution domain described in this application is a set composed of multiple heterogeneous feature distribution vectors. The heterogeneous feature distribution vectors in the heterogeneous feature distribution domain represent the vectors of the user in the abnormal behavior distribution feature dimension, which can reflect the diversity and complexity of the user's abnormal behavior.
[0092] In some embodiments, determining the multi-source heterogeneous gain entropy of the target shared document by using the preset data gain value and the heterogeneous feature distribution domain can be implemented by the following steps:
[0093] Performing feature extraction on all heterogeneous feature distribution vectors in the heterogeneous feature distribution domain to obtain a plurality of heterogeneous feature factors;
[0094] The multi-source heterogeneous gain entropy of the target shared document is determined according to all heterogeneous feature factors and preset data gain values.
[0095] It should be noted that the data gain value described in the present application represents an evaluation indicator of the contribution degree of the abnormal feature to the predictive ability of the target shared document anomaly during the abnormal feature selection process. The data gain value is used to evaluate the importance and contribution degree of the abnormal feature. In specific implementation, the data gain value can be preset based on historical experience.
[0096] In specific implementation, feature extraction is performed on all heterogeneous feature distribution vectors in the heterogeneous feature distribution domain to obtain multiple heterogeneous feature factors, which can be achieved in the following manner: first, a heterogeneous feature distribution vector is selected from the heterogeneous feature distribution domain, the mean and standard deviation of the heterogeneous feature distribution vector are calculated, the obtained mean is divided by twice the standard deviation, the value obtained by the division is negative and the base of the natural logarithm is taken, and the value obtained after taking the base of the natural logarithm is used as the heterogeneous feature factor, and then the above steps are repeated to obtain the remaining heterogeneous feature factors. Other methods can also be used in other embodiments, which are not limited here.
[0097] It should be noted that the heterogeneous feature factor described in this application represents an evaluation index of the importance of the heterogeneous feature distribution vector. The larger the heterogeneous feature factor, the higher the importance of the heterogeneous feature distribution vector corresponding to the heterogeneous feature factor in the overall heterogeneous feature distribution domain. The smaller the heterogeneous feature factor, the lower the importance of the heterogeneous feature distribution vector corresponding to the heterogeneous feature factor in the overall heterogeneous feature distribution domain.
[0098] In specific implementation, the multi-source heterogeneous gain entropy of the target shared document can be determined according to all heterogeneous feature factors and preset data gain values in the following manner, namely: first, all heterogeneous feature factors are divided by the data gain value, and the values obtained by the division are all taken as the logarithm with the base 10, and then all the values obtained by taking the logarithm are summed, and then the summed value is divided by the data gain value and the logarithm with the base 10 is taken, and finally, the value obtained by the division is used as the multi-source heterogeneous gain entropy of the target shared document. Other methods can also be used in other embodiments, which will not be repeated here.
[0099] It should be noted that the multi-source heterogeneous gain entropy described in the present application represents the degree of uncertainty of the abnormal information of the target shared document reduced by multi-source heterogeneous data, reflecting the effect of multi-source heterogeneous data in reducing the uncertainty of the target shared document information, so that these data can be selectively stored during storage to reduce the storage space occupied.
[0100] In step 104, the access heat records of users accessing the target shared document are extracted from the hot data area, and then the data disaster recovery coefficient of each user under extreme network conditions is determined based on the access heat records, and the disaster recovery segmented storage sequence of the target shared document is determined based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients.
[0101] It should be noted that the access heat refers to the access frequency of different types of data in the target shared document when the user accesses it; in specific implementation, the access times of text components, numeric components, table components, image components and link components in all multi-source heterogeneous hot data groups corresponding to each user are counted respectively, and the total number of text component accesses, numeric component accesses, table component accesses, image component accesses and link component accesses are counted to form a set, and the total number of accesses of each component in the set is used as the access heat value, and the obtained sets are all used as access heat sets, and each access heat set corresponds to a user, and the set composed of access heat sets is used as the access heat record of the user accessing the target shared document.
[0102] In some embodiments, reference Figure 3 As shown, this figure is a schematic diagram of the process of determining the data disaster recovery coefficient in some embodiments of the present application. In this embodiment, the data disaster recovery coefficient of each user under extreme network conditions is determined according to the access heat record, which can be implemented by the following steps:
[0103] First, in step 1041, the extreme network threshold of each user under extreme network conditions is determined according to the access heat record;
[0104] Next, in step 1042, a dissimilarity set of the target shared document is obtained;
[0105] Then, in step 1043, data disaster recovery is performed on all extreme network thresholds through the dissimilarity set to obtain a data disaster recovery coefficient for each user under extreme network conditions.
[0106] It should be noted that the extreme network conditions described in this application include high network latency, low network bandwidth, high network packet loss rate and network jitter;
[0107] In specific implementation, determining the extreme network threshold of each user under extreme network conditions according to the access heat record can be achieved in the following manner, namely: calculating the average of all access heat values in the access heat set of each user in the access heat record, and using the obtained average as the extreme network threshold of the corresponding user under extreme network conditions.
[0108] It should be noted that the extreme network threshold described in this application represents a reference value set according to the user's expected access level under extreme network conditions. According to the extreme network threshold, different storage measures can be taken to reduce the size of shared documents and increase download speed.
[0109] In specific implementation, data disaster recovery is performed on all extreme network thresholds through the dissimilarity set, and the data disaster recovery coefficient of each user under extreme network conditions can be obtained in the following manner, namely: first, the largest dissimilarity and the smallest dissimilarity are selected from the dissimilarity set, the smallest dissimilarity is subtracted from the largest dissimilarity, the subtracted value is divided by the mean of all dissimilarity in the dissimilarity set, and the divided value is used as the data dissimilarity characteristic value; then, all extreme network thresholds are divided by the data dissimilarity characteristic value, and the divided values are used as the data disaster recovery coefficients of the users corresponding to each extreme network threshold under extreme network conditions. Other methods can also be used in other embodiments, which are not limited here.
[0110] It should be noted that the data disaster recovery coefficient described in this application represents the parameter value of the recovery capability and protection level when the user's data is lost under extreme network conditions. The data disaster recovery coefficient can determine how to maximize the integrity and availability of data during the data storage process to reduce the risk of data in the face of disasters.
[0111] In some embodiments, determining the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients can be implemented by the following steps:
[0112] Perform linear fitting on all data disaster recovery coefficients to obtain a data disaster recovery curve;
[0113] Determine the disaster recovery segment amount of each user corresponding to each data disaster recovery coefficient according to the data disaster recovery curve;
[0114] The disaster recovery segment storage sequence of the target shared document is determined according to the multi-source heterogeneous gain entropy and all disaster recovery segment quantities.
[0115] It should be noted that in the present application, a linear fitting algorithm can be used to fit all data disaster recovery coefficients, for example: least squares method, polynomial fitting, least squares support vector machine and other fitting algorithms, and the fitted curve is used as the data disaster recovery curve. Other methods can also be used in other embodiments, which are not limited here.
[0116] The specific implementation is real-time. The disaster recovery segmentation amount of each user corresponding to each data disaster recovery coefficient is determined according to the data disaster recovery curve. It can be achieved in the following way, namely: the gradient is calculated for the position corresponding to each data disaster recovery coefficient of the data disaster recovery curve, and the gradient obtained at each position is used as the disaster recovery segmentation amount of the user corresponding to each data disaster recovery coefficient. Other methods can also be used in other embodiments, which are not limited here.
[0117] It should be noted that the disaster recovery segmentation amount described in the present application represents a quantitative value of dividing different types of data into different segments according to the degree of fluctuation of user data disaster recovery under extreme network conditions.
[0118] In specific implementation, the disaster recovery segment storage sequence of the target shared document is determined based on the multi-source heterogeneous gain entropy and all disaster recovery segment quantities, which can be achieved in the following manner, namely: first, the natural logarithm of the reciprocal of the multi-source heterogeneous gain entropy can be used as the adjustment coefficient of the disaster recovery segment quantity, and the product of the adjustment coefficient and each disaster recovery segment quantity can be used as the disaster recovery segment storage value. The disaster recovery segment storage value can be determined by the following formula, namely: disaster recovery segment storage value = adjustment coefficient * disaster recovery segment quantity, and then, all disaster recovery segment storage values are sorted in ascending order, and the sorted sequence is used as the disaster recovery segment storage sequence of the target shared document. It should be noted that each disaster recovery segment storage value in the present application corresponds to a type of data, and other methods can also be used in other embodiments, which will not be repeated here.
[0119] In step 105, hierarchical storage processing is performed on the data of the diversified data sources of the target shared document according to the disaster recovery segment storage sequence.
[0120] In some embodiments, hierarchical storage processing of data from diversified data sources of target shared documents according to the disaster recovery segment storage sequence can be implemented by the following steps:
[0121] Determine hierarchical storage coefficients for different types of data from a target shared document diversified data source according to the disaster recovery segment storage sequence;
[0122] The data corresponding to the diversified data sources of the target shared document are stored and processed through all hierarchical storage coefficients.
[0123] In a specific implementation, determining the hierarchical storage coefficients of different types of data of the diversified data sources of the target shared document according to the disaster recovery segment storage sequence can be implemented in the following manner, namely: first, the disaster recovery segment storage sequence is divided into three equal parts in order of size, and the sequence formed by the first part of the disaster recovery segment storage sequence is used as a high-level hierarchical sequence, the sequence formed by the middle part of the disaster recovery segment storage sequence is used as a medium-level hierarchical sequence, and the sequence formed by the last part of the disaster recovery segment storage sequence is used as a low-level hierarchical sequence. For example, if the disaster recovery segment storage sequence is [1,2,3,4,5,6], the high-level hierarchical sequence obtained after three equal parts is [1,2], the medium-level hierarchical sequence is [3,4], and the low-level hierarchical sequence is [1,2]. Hierarchical sequence [5,6], if the total number of disaster recovery segment storage values in the disaster recovery segment storage sequence is an odd number, the last one in the disaster recovery segment storage sequence is divided into a low-level hierarchical sequence, and then, the hierarchical storage coefficient of the data type corresponding to the disaster recovery segment storage value in the high-level hierarchical sequence is set to 1, the hierarchical storage coefficient of the data type corresponding to the disaster recovery segment storage value in the intermediate hierarchical sequence is set to 2, and the hierarchical storage coefficient of the data type corresponding to the disaster recovery segment storage value in the low-level hierarchical sequence is set to 3. It should be noted that the hierarchical storage coefficient described in the present application can help determine the storage priority of different types of data, and other methods can also be used in other embodiments, which is not limited here.
[0124] In specific implementation, the storage processing of the data corresponding to the target shared document diversified data source through all hierarchical storage coefficients can be achieved in the following manner, namely: if the hierarchical storage coefficient of the data corresponding to the target shared document diversified data source is equal to 1, the type data corresponding to the hierarchical storage coefficient is stored on the storage medium with the highest performance, such as a solid state drive (SSD), which can ensure that high-level data can be quickly accessed and responded to; if the hierarchical storage coefficient of the data corresponding to the target shared document diversified data source is equal to 2, the type data corresponding to the hierarchical storage coefficient is stored on a storage medium with moderate performance, such as an enterprise-level hard disk array (Serial Attached SCSI, SAS) or a high-speed hard disk; if the hierarchical storage coefficient of the data corresponding to the target shared document diversified data source is equal to 3, the type data corresponding to the hierarchical storage coefficient is stored on a storage medium with lower cost, such as a large-capacity disk array (Hard Disk Drive, HDD) or a cold storage service.
[0125] In addition, in another aspect of the present application, in some embodiments, the present application provides a data processing system for sharing a diversified data source of a document, referring to Figure 4, which is a schematic diagram of exemplary hardware and / or software of a data processing system for sharing a diversified data source of a document according to some embodiments of the present application, the data processing system 400 for sharing a diversified data source of a document includes: an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows:
[0126] Acquisition module 401, in this application, acquisition module 401 is mainly used to acquire data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data;
[0127] Processing module 402, in the present application, is used to partition the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas;
[0128] It should be noted that the processing module 402 in the present application is also used to perform anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determine the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determine the multi-source heterogeneous gain entropy of the target shared document through the preset data gain value and the heterogeneous feature distribution domain;
[0129] In addition, it should be noted that the processing module 402 in the present application is also used to extract the access heat record of the user accessing the target shared document from the hot data area, and then determine the data disaster recovery coefficient of each user under extreme network conditions based on the access heat record, and determine the disaster recovery segmented storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients;
[0130] Execution module 403, in the present application, execution module 403 is mainly used to perform hierarchical storage processing on the data of the diversified data sources of the target shared document according to the disaster recovery segment storage sequence.
[0131] In addition, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned data processing method for shared document diversified data sources.
[0132] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device according to some embodiments of the present application using a data processing method for a shared document with diversified data sources. The data processing method for a shared document with diversified data sources in the above embodiments can be performed by Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.
[0133] The processor 501 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more processors for controlling the execution of the data processing method for sharing a diversified data source of a document in the present application.
[0134] The communication bus 502 may be used to transmit information between the above-mentioned components.
[0135] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.
[0136] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The method described in the above method embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.
[0137] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0138] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0139] The above-mentioned computer device can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.
[0140] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the data processing method for the shared document diversified data source is implemented.
[0141] In summary, in the data processing method and system of the shared document diversified data source disclosed in the embodiment of the present application, data from different data sources are obtained from the target shared document, thereby obtaining multi-source heterogeneous data; the multi-source heterogeneous data is partitioned according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas; different types of data in the cold data area are detected for abnormality to obtain heterogeneous abnormal data, and then the heterogeneous feature distribution domain of the user in the process of accessing the target shared document is determined according to the heterogeneous abnormal data, and the multi-source heterogeneous gain entropy of the target shared document is determined by the preset data gain value and the heterogeneous feature distribution domain; the access heat record of the user accessing the target shared document is extracted from the hot data area, and then the data disaster recovery coefficient of each user under extreme network conditions is determined according to the access heat record, and the disaster recovery segmented storage sequence of the target shared document is determined based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients; the data of the diversified data sources of the target shared document are hierarchically stored according to the disaster recovery segmented storage sequence. The scheme of the present application can realize data disaster recovery for the data of the server storage node under extreme network conditions, thereby improving the stability of the server storing diversified data.
[0142] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0143] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data processing method for sharing diversified data sources of documents, It is characterized in that The steps include: Acquire data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data; Partitioning the multi-source heterogeneous data according to user access characteristics to different types of data to obtain cold data areas and hot data areas; Performing anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determining the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determining the multi-source heterogeneous gain entropy of the target shared document through a preset data gain value and the heterogeneous feature distribution domain; Extracting the access heat records of users accessing the target shared document from the hot data area, and then determining the data disaster recovery coefficient of each user under extreme network conditions according to the access heat records, and determining the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all the data disaster recovery coefficients; The data of the diversified data sources of the target shared document are hierarchically stored according to the disaster recovery segment storage sequence.
2. The method according to claim 1, It is characterized in that Partitioning the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas specifically includes: Obtain user access characteristics to different types of data; Mapping the multi-source heterogeneous data through the access feature to obtain multi-source heterogeneous mapping data; Based on a pre-trained hot and cold data classification model, the access features are marked as hot and cold data to obtain hot and cold data access features; The multi-source heterogeneous mapping data is partitioned into cold and hot data according to the cold and hot data access characteristics to obtain a cold data area and a hot data area.
3. The method according to claim 1, It is characterized in that Anomaly detection is performed on different types of data in the cold data area to obtain heterogeneous anomaly data, specifically including: Obtain user access characteristics to different types of data; Extracting access time series of different types of data by users in the cold data area through the access features; Determining a plurality of abnormal data marking points in the cold data area according to the access time series; Determine the abnormal data critical value based on all abnormal data marking points; The heterogeneous abnormal data in the cold data area is determined by the abnormal data critical value.
4. The method according to claim 1, It is characterized in that Determining the multi-source heterogeneous gain entropy of the target shared document by using the preset data gain value and the heterogeneous feature distribution domain specifically includes: Performing feature extraction on all heterogeneous feature distribution vectors in the heterogeneous feature distribution domain to obtain a plurality of heterogeneous feature factors; The multi-source heterogeneous gain entropy of the target shared document is determined according to all heterogeneous feature factors and preset data gain values.
5. The method according to claim 1, It is characterized in that Determining the data disaster recovery coefficient of each user under extreme network conditions based on the access heat record specifically includes: Determining an extreme network threshold of each user under extreme network conditions according to the access heat record; Obtaining a dissimilarity set of the target shared document; Data disaster recovery is performed on all extreme network thresholds through the dissimilarity set to obtain a data disaster recovery coefficient for each user under extreme network conditions.
6. The method according to claim 1, It is characterized in that Determining the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients specifically includes: Perform linear fitting on all data disaster recovery coefficients to obtain a data disaster recovery curve; Determine the disaster recovery segment amount of each user corresponding to each data disaster recovery coefficient according to the data disaster recovery curve; The disaster recovery segment storage sequence of the target shared document is determined according to the multi-source heterogeneous gain entropy and all disaster recovery segment quantities.
7. The method according to claim 1, It is characterized in that The hierarchical storage processing of the data of the target shared document diversified data sources according to the disaster recovery segment storage sequence specifically includes: Determine hierarchical storage coefficients for different types of data from a target shared document diversified data source according to the disaster recovery segment storage sequence; The data corresponding to the diversified data sources of the target shared document are stored and processed through all hierarchical storage coefficients.
8. A data processing system for sharing diversified data sources of documents, It is characterized in that include: The acquisition module is used to acquire data from different data sources from the target shared document, thereby obtaining multi-source heterogeneous data; A processing module, used for partitioning the multi-source heterogeneous data according to the user's access characteristics to different types of data to obtain cold data areas and hot data areas; The processing module is further used to perform anomaly detection on different types of data in the cold data area to obtain heterogeneous anomaly data, and then determine the heterogeneous feature distribution domain of the user in the process of accessing the target shared document according to the heterogeneous anomaly data, and determine the multi-source heterogeneous gain entropy of the target shared document through the preset data gain value and the heterogeneous feature distribution domain; The processing module is further used to extract the access heat records of users accessing the target shared document from the hot data area, and then determine the data disaster recovery coefficient of each user under extreme network conditions according to the access heat records, and determine the disaster recovery segment storage sequence of the target shared document based on the multi-source heterogeneous gain entropy and all data disaster recovery coefficients; The execution module is used to perform hierarchical storage processing on the data of the diversified data sources of the target shared document according to the disaster recovery segment storage sequence.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the data processing method for sharing a diversified data source of a document according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the data processing method for sharing a diversified data source of a document as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Optimal distribution method and device for communication data transmission bandwidth of WSNs of intelligent power distribution network
CN113038538A
Cold and hot data separation method and device based on big data, equipment and medium
CN116150184A
