Collaborative analysis method of archival data based on deep learning
Through the deep learning-based archival data collaborative analysis method, the problems of large amount of data and complex structure in archival data analysis are solved, more accurate analysis and decision-making support are achieved, and data utilization efficiency and reliability are improved.
Patent Information
- Application Number
- CN202410693047.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-31
AI Technical Summary
During the analysis of archive data, the data volume is large and unstable, the computing resources are consumed, and the data structure is complex, which makes it difficult to integrate data and cannot conduct accurate analysis. The analysis process is cumbersome, which affects the progress of work.
Adopt the archival data collaborative analysis method based on deep learning, and obtain target archival data, divide the entire data area, build archive node trees, extract features, form comprehensive feature vectors, use cluster analysis and entity recognition to perform collaborative analysis, and obtain collaborative analysis deviations. to handle abnormal situations and improve analysis accuracy.
It realizes a more comprehensive and accurate analysis of archival data, discovers potential problems and opportunities, provides effective decision-making support, improves data utilization efficiency and accuracy, and enhances data reliability and usability.
Smart Images

Figure CN118395124B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a collaborative analysis method of archival data based on deep learning. Background Art
[0002] Deep learning is a machine learning method that learns and understands complex data patterns by building a neural network with multiple layers. Data collaborative analysis is a method that uses multiple data sources and multiple analysis techniques for comprehensive analysis.
[0003] Data collaborative analysis methods are often applied in various fields, such as business intelligence and data analysis, finance and risk management, urban planning and smart cities, manufacturing and supply chain management, etc. At present, in the process of archival data analysis, by integrating data from different sources and different types, and using statistical analysis, machine learning, data mining and other methods to discover the associations and patterns between data, efficient analysis of archival data can be achieved.
[0004] However, in the process of data analysis, the acquired archival data may be too large or unstable. In addition, the computing resources are consumed a lot, and the data has a complex structure and association, which limits the integration of data resources. It is impossible to accurately analyze all archival data and it is challenging, resulting in the data that needs to be processed and analyzed being chaotic and cumbersome, delaying the progress of work. Summary of the invention
[0005] The purpose of the present invention is to solve the problems in the background technology and to propose a collaborative analysis method of archival data based on deep learning.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] The collaborative analysis method of archival data based on deep learning includes:
[0008] Step 1: Obtain target archive data, divide the entire data domain into different attribute learning areas, and then obtain corresponding learning sub-areas;
[0009] Step 2: construct an archive node tree for the learning sub-area, wherein the archive node tree includes a root node, a parent node, and a child node;
[0010] Step 3: Establish relationships between the root node, parent node, and child node of the archive node tree in combination with the hierarchical structure and permutation combination of the node tree;
[0011] Step 4: Extract features from the data of each learning sub-region and count the number of sub-nodes in all learning sub-regions;
[0012] Step 5: Introduce a matrix vector, collect the acquired text feature vector, image feature vector and video feature vector, and combine the three feature vectors to form a comprehensive feature vector of the learning sub-region;
[0013] Step 6: Use a cluster analysis algorithm to obtain identification values, and process and analyze the identification values to obtain a first deep learning set;
[0014] Step 7: perform entity recognition and extraction on text information, image information, and video information according to the query information and operation records of the archive data, obtain entity segments, and process and analyze the entity segments to obtain a second deep learning set;
[0015] Step 8: According to the collaborative analysis of the first deep learning set and the second deep learning set, obtain the collaborative analysis deviation, and further process and analyze the collaborative analysis deviation.
[0016] It should be noted that the deep learning-based collaborative analysis method of archival data proposed in the present invention can make full use of the interrelationships and complementarities between data, provide more comprehensive and accurate analysis results and decision support, help users better understand the information behind the data, discover potential problems and opportunities, and provide effective solutions.
[0017] Furthermore, the process of obtaining target archive data, dividing the entire data domain into different attribute learning areas, and then obtaining corresponding learning sub-areas includes:
[0018] Obtain target archive data and divide the data into a whole domain; wherein the whole domain includes a first attribute learning area, a second attribute learning area, a third attribute learning area, ..., an mth attribute learning area; m represents the number of divided attribute learning areas;
[0019] All attribute learning regions are marked as SXQ; various attribute learning regions represent target archive data of different attribute categories, and each attribute category corresponds to a learning region;
[0020] Each attribute learning region includes at least one learning sub-region ; Subdivide the learning sub-area according to the stored file format; wherein w, p and v represent the text learning sub-area, image learning sub-area and video learning sub-area respectively.
[0021] Furthermore, the process of constructing the archive node tree for the learning sub-area includes:
[0022] Create corresponding root nodes according to different storage formats of learning sub-areas;
[0023] Creating a number of parent nodes for the root node;
[0024] Create corresponding child nodes based on several parent nodes.
[0025] Furthermore, the process of establishing a relationship between the root node, the parent node, and the child node of the archive node tree in combination with the hierarchical structure and the permutation and combination of the node tree includes:
[0026] If you learn sub-areas , then the learning sub-region is a text learning sub-region, the text data of the text learning sub-region is taken as the root node, and the text parameters of the text data are recorded as the parent node, and the text type, keywords, index description, page range, date and text size of the text parameters are recorded as child nodes;
[0027] If you learn sub-areas , then the learning sub-region is an image learning sub-region, the image data of the image learning sub-region is taken as the root node, the image parameters of the image data are recorded as the parent node, and the project type, date and image size of the image parameters are recorded as child nodes;
[0028] If you learn sub-areas , then the learning sub-area is a video learning sub-area, the video data of the video learning sub-area is taken as the root node, the video parameters of the video data are recorded as the parent node, and the project type, date and video size of the video parameters are recorded as child nodes.
[0029] Furthermore, the process of extracting features from the data of each learning sub-region and counting the number of sub-nodes of all learning sub-regions includes:
[0030] Extract features from subnodes of different learning sub-areas to obtain text features, image features, and video features; obtain the number of subnodes corresponding to text features. , child node data packet of image features And the subnode data packet of the video feature ;
[0031] Among them, d represents the number of the sub-node quantity package of the text feature, and d=1, 2,..., n; n represents the total number of the sub-node quantity packages of the text feature; f represents the number of the sub-node data package of the image feature, and f=1, 2,..., k; k represents the total number of the sub-node data packages of the image feature; g represents the number of the sub-node data package of the video feature, and g=1, 2,..., j; j represents the total number of the sub-node data packages of the video feature.
[0032] Furthermore, the process of introducing a matrix vector, collecting the acquired text feature vector, image feature vector and video feature vector, and combining the three feature vectors to form a comprehensive feature vector of the learning sub-region includes:
[0033] Introduce matrix-vector notation based on data feature extraction and child node number statistics:
[0034] , , ,
[0035] The comprehensive feature vector X of the learning sub-region is calculated using the formula; where, ; In the formula, a is the development coefficient, bi is the support coefficient, D is the support item, r is the support index of the support item D, and R is the preset proportional coefficient of different eigenvectors; i=1, 2,..., u; and 0<u<1, r is a real number greater than 0.
[0036] Furthermore, a cluster analysis algorithm is used to obtain identification values, and a process of processing and analyzing the identification values to obtain a first deep learning set includes:
[0037] The obtained comprehensive feature vector X is mapped into an identification value SY using the clustering analysis algorithm L: SY=L(X); wherein the clustering analysis algorithm selects the K-Means algorithm;
[0038] It should be noted that the K-Means algorithm is a centroid-based clustering algorithm, which can divide data points into K clusters and assign data points to the nearest centroid, thereby realizing data clustering; in the K-Means algorithm, the centroid is the center point of each cluster, representing the average position of all data points in the cluster, and the position of the centroid changes dynamically, which is the core of clustering and is used to determine the cluster to which the data point belongs; the K-Means algorithm assigns data points to the nearest centroid through continuous iteration, and updates the position of the centroid until the position of the centroid no longer changes; in an embodiment of the present invention, the acquired comprehensive feature vector X is mapped to an identification value SY, which is determined by the K-Means algorithm and represents the cluster to which the comprehensive feature vector X belongs. Each cluster has a fixed identification value, which is an integer representing the cluster number; therefore, the identification value SY is a fixed value, indicating the cluster number to which the comprehensive feature vector X belongs, and this number is calculated by the K-Means algorithm based on the characteristics and distance of the data points;
[0039] By adding the timestamp to the identification value SY, a unique identification value SY0 of the learning sub-area is formed;
[0040] Among them, by adding a timestamp to the identification value to obtain a unique identification value, it is ensured that the exact time point of generation can be traced back according to each unique identification value, which is helpful to track and identify the identification values generated at different time points in the collaborative analysis method of archival data;
[0041] A set of unique identification values of each learning sub-area is obtained and merged into a first deep learning set.
[0042] Furthermore, the process of performing entity recognition and extraction on text information, image information, and video information according to the query information and operation records of the archive data, obtaining entity segments, and processing and analyzing the entity segments to obtain a second deep learning set includes:
[0043] Extract entity and relationship information from text, images, and video information to obtain single or multiple triples;
[0044] The extracted entity and relationship information is stored in a learning triple set Trip, where the elements are triples in the form of (c, h, t), where c, h, and t represent text elements, image elements, and video elements, respectively.
[0045] Count the query times xs and operation frequency zp of archive data in the database;
[0046] generating entity segments based on data in the set of learned triples;
[0047] It should be noted that an entity segment is a part with entity meaning extracted from different types of information such as text, images and videos. In the process of information extraction and entity recognition, it refers to a fragment or part that is recognized and considered to have a specific entity meaning, which can be a word, phrase, sentence, a specific object or scene in an image, or a specific object, action or scene in a video. Take an example of an entity segment: in text information, a character, word or phrase in the stored information can be identified as an entity segment, such as "apple", "iPhone 15", "2024 Annual Report", etc.; in image information, a specific object, person or scene can be identified as an entity segment, such as "car", "tree", "street light" in a photo, etc.; in video information, a specific object, action or scene can be identified as an entity segment, such as "pedestrians crossing the road", "plane taking off", "basketball game", etc. in a video; the recognition and extraction of entity segments facilitates the structured processing and analysis of different types of information such as text, images and videos, so as to better understand the data content and support subsequent information mining and analysis.
[0048] Use the generated entity blocks to construct ternary learning relations to obtain the ternary learning index ; Among them, the expression of the ternary learning index is: In the formula, ℇ represents the standard variation parameter of the entity segment, are different ternary weight ratio coefficients respectively;
[0049] Set unique reference values for each entity segment through ternary learning relationship expressions , and merge to obtain a second deep learning set;
[0050] Among them, the only benchmark value The steps of obtaining include: determining the specific values of c, h, and t; calculating based on the given data The value of Substitute the value of into the expression, according to the given weight coefficient Calculate; repeat the above steps, sort all ternary learning indexes in descending order according to the calculated values, remove the maximum and minimum values and take the average to obtain a unique benchmark value .
[0051] Furthermore, according to the collaborative analysis of the first deep learning set and the second deep learning set, the process of obtaining the collaborative analysis deviation and further processing and analyzing the collaborative analysis deviation includes:
[0052] Obtain a first deep learning set H1 and a second deep learning set H2; integrate the first deep learning set H1 and the second deep learning set H2 to calculate a collaborative analysis deviation PL; wherein the formula for the collaborative analysis deviation is: ; In the formula, and Respectively represent the entity segments or learning sub-areas in the first deep learning set and the second deep learning set, and the function It is an indicator or measurement method to measure the relationship between two entity segments. The integral symbol ∫ indicates the integration of all entity segments or learning sub-regions;
[0053] It should be noted that multiple entity segments constitute a learning entity block, and a learning entity block corresponds to a learning sub-area; in the integral formula, the function It may include differences, similarities or other correlation measures between a series of features, and different methods can be used to calculate according to needs, for example, using Euclidean distance, correlation coefficient, cosine similarity or other measurement methods to measure the degree of deviation between two features; it should be noted that the above formula for the deviation of collaborative analysis is only a framework, and the specific calculation method and measurement indicators need to be selected and defined according to the actual situation and data characteristics. Therefore, in practical applications, the formula needs to be adjusted and optimized according to the specific situation;
[0054] Setting a critical threshold of the collaborative analysis deviation; comparing each collaborative analysis deviation obtained with the preset critical threshold;
[0055] Screen out the collaborative analysis deviations that are less than a preset critical threshold, and count the number G of the screened collaborative analysis deviations; if G is less than or equal to 1, use the first deep learning set corresponding to the screened collaborative analysis deviations as the optimal collaborative analysis solution; if G is greater than 1, clear the first deep learning set, automatically generate a clearing signal, and reacquire the first deep learning set, and repeat the above operation;
[0056] Among them, the above analysis process can ensure that abnormal situations in the archival data set are handled in a timely manner to improve the accuracy and reliability of the collaborative analysis of archival data; through the above steps, abnormal situations can be discovered and handled in a timely manner during the collaborative analysis process, thereby ensuring the effectiveness and credibility of the archival data analysis results.
[0057] Compared with the existing technology, the advantages of the archival data collaborative analysis method based on deep learning provided by the present invention are:
[0058] The present invention collects target archive data and divides the data according to different attributes or characteristics to form various learning areas, thereby preparing for subsequent analysis and processing; analyzing and processing the structure of the archive node tree, and establishing the relationship between different nodes for subsequent data analysis and mining;
[0059] The present invention extracts features from the data of each learning sub-region for subsequent analysis and modeling; at the same time, counts the number of sub-nodes in each learning sub-region to provide basic information for subsequent data processing and analysis; summarizes and integrates the feature vectors of the acquired text, image and video data to form a comprehensive feature vector for each learning sub-region;
[0060] The present invention performs entity recognition and extraction on text, image and video information according to query information and operation records, thereby obtaining entity segments, and processes and analyzes the entity segments to form a second deep learning set to provide a reference basis for subsequent decision-making.
[0061] The present invention performs collaborative analysis through the first deep learning set and the second deep learning set to obtain a collaborative analysis deviation, and further processes and analyzes the collaborative analysis deviation to obtain deeper insights and understanding.
[0062] In summary, the present invention can verify the collaborative analysis method of archival data according to actual conditions, realize information sharing and collaborative work, improve the utilization efficiency and accuracy of data, and through comprehensive data analysis, involving data partitioning, feature extraction, clustering analysis, and generation and analysis of deep learning sets, it helps to enhance the reliability and availability of data, provide reliable support and guarantee, and ensure the normal implementation of subsequent collaborative analysis methods of archival data based on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 This is a flow chart of the deep learning-based collaborative analysis method for archival data proposed in the present invention. DETAILED DESCRIPTION
[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0065] Reference Figure 1 , a collaborative analysis method for archival data based on deep learning, including:
[0066] Step 1: Obtain target archive data, divide the entire data domain into different attribute learning areas, and then obtain corresponding learning sub-areas;
[0067] Step 2: construct an archive node tree for the learning sub-area, wherein the archive node tree includes a root node, a parent node, and a child node;
[0068] Step 3: Establish relationships between the root node, parent node, and child node of the archive node tree in combination with the hierarchical structure and permutation combination of the node tree;
[0069] Step 4: Extract features from the data of each learning sub-region and count the number of sub-nodes in all learning sub-regions;
[0070] Step 5: Introduce a matrix vector, collect the acquired text feature vector, image feature vector and video feature vector, and combine the three feature vectors to form a comprehensive feature vector of the learning sub-region;
[0071] Step 6: Use a cluster analysis algorithm to obtain identification values, and process and analyze the identification values to obtain a first deep learning set;
[0072] Step 7: perform entity recognition and extraction on text information, image information, and video information according to the query information and operation records of the archive data, obtain entity segments, and process and analyze the entity segments to obtain a second deep learning set;
[0073] Step 8: According to the collaborative analysis of the first deep learning set and the second deep learning set, obtain the collaborative analysis deviation, and further process and analyze the collaborative analysis deviation.
[0074] It should be noted that the deep learning-based collaborative analysis method of archival data proposed in the present invention can make full use of the interrelationships and complementarities between data, provide more comprehensive and accurate analysis results and decision support, help users better understand the information behind the data, discover potential problems and opportunities, and provide effective solutions.
[0075] In step 1, the process of obtaining target archive data, dividing the entire data domain to obtain different attribute learning areas, and then obtaining corresponding learning sub-areas includes:
[0076] S101, obtaining target archive data, and dividing the data into a whole domain; wherein the data whole domain includes a first attribute learning area, a second attribute learning area, a third attribute learning area, ..., an mth attribute learning area; m represents the number of divided attribute learning areas;
[0077] S102, marking all attribute learning areas as SXQ; wherein various attribute learning areas represent target archive data of different attribute categories, and each attribute category corresponds to a learning area:
[0078] S103: Each attribute learning area includes at least one learning sub-area ; Subdivide the learning sub-area according to the stored file format; wherein w, p and v represent the text learning sub-area, image learning sub-area and video learning sub-area respectively.
[0079] In the step 2, the process of constructing the archive node tree for the learning sub-area includes:
[0080] S201, creating corresponding root nodes according to different storage formats of learning sub-areas;
[0081] S202, creating several parent nodes for the root node;
[0082] S203: Create corresponding child nodes based on several parent nodes.
[0083] In step 3, the process of establishing a relationship between the root node, the parent node, and the child node of the archive node tree in combination with the hierarchical structure and permutation combination of the node tree includes:
[0084] S301, if learning sub-area , then the learning sub-region is a text learning sub-region, the text data of the text learning sub-region is taken as the root node, and the text parameters of the text data are recorded as the parent node, and the text type, keywords, index description, page range, date and text size of the text parameters are recorded as child nodes;
[0085] S302: If the sub-area is learned , then the learning sub-region is an image learning sub-region, the image data of the image learning sub-region is taken as the root node, the image parameters of the image data are recorded as the parent node, and the project type, date and image size of the image parameters are recorded as child nodes;
[0086] S303: If the sub-area is learned , then the learning sub-area is a video learning sub-area, the video data of the video learning sub-area is taken as the root node, the video parameters of the video data are recorded as the parent node, and the project type, date and video size of the video parameters are recorded as child nodes.
[0087] In the step 4, the process of extracting features from the data of each learning sub-region and counting the number of sub-nodes of all learning sub-regions includes:
[0088] S401, extracting features from sub-nodes of different learning sub-areas to obtain text features, image features, and video features respectively; obtaining the number of sub-nodes corresponding to the text features, including , child node data packet of image features And the subnode data packet of the video feature ;
[0089] Among them, d represents the number of the sub-node quantity package of the text feature, and d=1, 2,..., n; n represents the total number of the sub-node quantity packages of the text feature; f represents the number of the sub-node data package of the image feature, and f=1, 2,..., k; k represents the total number of the sub-node data packages of the image feature; g represents the number of the sub-node data package of the video feature, and g=1, 2,..., j; j represents the total number of the sub-node data packages of the video feature.
[0090] In step 5, the process of introducing a matrix vector, collecting the acquired text feature vector, image feature vector and video feature vector, and combining the three feature vectors to form a comprehensive feature vector of the learning sub-region includes:
[0091] S501, based on data feature extraction and sub-node number statistics results, matrix vector notation is introduced:
[0092] , , , the comprehensive feature vector X of the learning sub-region is calculated using the formula; where, ; In the formula, a is the development coefficient, bi is the support coefficient, D is the support item, r is the support index of the support item D, and R is the preset proportional coefficient of different eigenvectors; i=1, 2,..., u; and 0<u<1, r is a real number greater than 0.
[0093] In step 6, the process of using a cluster analysis algorithm to obtain an identification value and processing and analyzing the identification value to obtain a first deep learning set includes:
[0094] S601, using a cluster analysis algorithm L to map the obtained comprehensive feature vector X into an identification value SY: SY=L(X); wherein the cluster analysis algorithm selects a K-Means algorithm;
[0095] It should be noted that the K-Means algorithm is a centroid-based clustering algorithm, which can divide data points into K clusters and assign data points to the nearest centroid, thereby realizing data clustering; in the K-Means algorithm, the centroid is the center point of each cluster, representing the average position of all data points in the cluster, and the position of the centroid changes dynamically, which is the core of clustering and is used to determine the cluster to which the data point belongs; the K-Means algorithm assigns data points to the nearest centroid through continuous iteration, and updates the position of the centroid until the position of the centroid no longer changes; in an embodiment of the present invention, the acquired comprehensive feature vector X is mapped to an identification value SY, which is determined by the K-Means algorithm and represents the cluster to which the comprehensive feature vector X belongs. Each cluster has a fixed identification value, which is an integer representing the cluster number; therefore, the identification value SY is a fixed value, indicating the cluster number to which the comprehensive feature vector X belongs, and this number is calculated by the K-Means algorithm based on the characteristics and distance of the data points;
[0096] S602, by adding the timestamp to the identification value SY, a unique identification value SY0 of the learning sub-area is formed;
[0097] Among them, by adding a timestamp to the identification value to obtain a unique identification value, it is ensured that the exact time point of generation can be traced back according to each unique identification value, which is helpful to track and identify the identification values generated at different time points in the collaborative analysis method of archival data;
[0098] S603: Obtain a set of unique identification values for each learning sub-area, and merge them into a first deep learning set.
[0099] In step seven, the process of performing entity recognition and extraction on text information, image information, and video information according to the query information and operation records of the archive data, obtaining entity segments, and processing and analyzing the entity segments to obtain the second deep learning set includes:
[0100] S701, extracting entity and relationship information from text, image and video information to obtain a single or any number of triples;
[0101] S702, storing the extracted entity and relationship information into a learning triple set Trip; the elements of which are triples in the form of (c, h, t); wherein c, h and t represent text elements, image elements and video elements respectively;
[0102] S703, counting the query times xs and operation frequency zp of archive data in the database;
[0103] S704, generating entity segments according to the data in the learning triple set;
[0104] It should be noted that an entity segment is a part with entity meaning extracted from different types of information such as text, images and videos. In the process of information extraction and entity recognition, it refers to a fragment or part that is recognized and considered to have a specific entity meaning, which can be a word, phrase, sentence, a specific object or scene in an image, or a specific object, action or scene in a video. Take an example of an entity segment: in text information, a character, word or phrase in the stored information can be identified as an entity segment, such as "apple", "iPhone 15", "2024 Annual Report", etc.; in image information, a specific object, person or scene can be identified as an entity segment, such as "car", "tree", "street light" in a photo, etc.; in video information, a specific object, action or scene can be identified as an entity segment, such as "pedestrians crossing the road", "plane taking off", "basketball game", etc. in a video; the recognition and extraction of entity segments facilitates the structured processing and analysis of different types of information such as text, images and videos, so as to better understand the data content and support subsequent information mining and analysis.
[0105] S705: Use the generated entity blocks to construct a ternary learning relationship to obtain a ternary learning index ; Among them, the expression of the ternary learning index is: ; where ℇ represents the standard variation parameter of the entity segment, are different ternary weight ratio coefficients respectively;
[0106] S706: Setting a unique reference value for each entity segment through a ternary learning relationship expression , and merge to obtain a second deep learning set;
[0107] Among them, the only benchmark value The steps of obtaining include: determining the specific values of c, h, and t; calculating based on the given data The value of Substitute the value of into the expression, according to the given weight coefficient Calculate; repeat the above steps, sort all ternary learning indexes in descending order according to the calculated values, remove the maximum and minimum values and take the average to obtain a unique benchmark value .
[0108] In step eight, the process of obtaining the collaborative analysis deviation according to the collaborative analysis of the first deep learning set and the second deep learning set, and further processing and analyzing the collaborative analysis deviation includes:
[0109] S801, obtaining a first deep learning set H1 and a second deep learning set H2; integrating the first deep learning set H1 and the second deep learning set H2 to calculate a collaborative analysis deviation PL; wherein the formula for the collaborative analysis deviation is: ; In the formula, and Respectively represent the entity segments or learning sub-areas in the first deep learning set and the second deep learning set, and the function It is an indicator or measurement method to measure the relationship between two entity segments. The integral symbol ∫ indicates the integration of all entity segments or learning sub-regions;
[0110] It should be noted that multiple entity segments constitute a learning entity block, and a learning entity block corresponds to a learning sub-area; in the integral formula, the function It may include differences, similarities or other correlation measures between a series of features, and different methods can be used to calculate according to needs, for example, using Euclidean distance, correlation coefficient, cosine similarity or other measurement methods to measure the degree of deviation between two features; it should be noted that the above formula for the deviation of collaborative analysis is only a framework, and the specific calculation method and measurement indicators need to be selected and defined according to the actual situation and data characteristics. Therefore, in practical applications, the formula needs to be adjusted and optimized according to the specific situation;
[0111] S802, setting a critical threshold of the collaborative analysis deviation; comparing each acquired collaborative analysis deviation with the preset critical threshold;
[0112] S803, filter out collaborative analysis deviations that are less than a preset critical threshold, and count the number G of the filtered collaborative analysis deviations; if G is less than or equal to 1, use the first deep learning set corresponding to the filtered collaborative analysis deviation as the optimal collaborative analysis solution; if G is greater than 1, clear the first deep learning set, automatically generate a clear signal, and reacquire the first deep learning set, and repeat the above operation;
[0113] Among them, the above analysis process can ensure that abnormal situations in the archival data set are handled in a timely manner to improve the accuracy and reliability of the collaborative analysis of archival data; through the above steps, abnormal situations can be discovered and handled in a timely manner during the collaborative analysis process, thereby ensuring the effectiveness and credibility of the archival data analysis results.
[0114] In the embodiment of the present invention, by collecting the target archive data and dividing the data according to different attributes or features, various learning areas are formed to prepare for subsequent analysis and processing, by analyzing and processing the structure of the archive node tree, the relationship between different nodes is established for subsequent data analysis and mining, by counting the number of sub-nodes in each learning sub-area, basic information is provided for subsequent data processing and analysis, by summarizing and integrating the feature vectors of the acquired text, image and video data, a comprehensive feature vector of each learning sub-area is formed, by processing and analyzing the entity segment, a second deep learning set is formed, a reference basis is provided for subsequent decision-making, and the quality and reliability of the data are ensured, and the first deep learning set and the second deep learning set are collaboratively analyzed to obtain the collaborative analysis deviation, and further processed and analyzed to obtain a deeper level of insight and understanding. In summary, the example of the present invention involves the decision-making of data processing, feature extraction and comprehensive analysis, and solves the technical problem that the existing scheme does not perform deep learning and collaborative analysis on the archive data, resulting in poor analysis of the archive data. In actual situations, more data and context information may be needed to make specific decisions and optimization plans.
[0115] In addition, the formulas involved in the above are all calculated by removing dimensions and taking their numerical values. They are a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The proportional coefficient in the formula and the various preset thresholds in the analysis process are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data; the size of the proportional coefficient is to quantify each parameter to obtain a specific value for subsequent comparison. The size of the proportional coefficient depends on the amount of sample data and the preliminary setting of the corresponding processing coefficient for each group of sample data by technical personnel in this field; as long as it does not affect the proportional relationship between the parameter and the quantized value.
[0116] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically based on the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0117] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0118] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0122] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;
[0123] Finally: The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A collaborative analysis method of archival data based on deep learning, characterized by: Step 1: Obtain target archive data, divide the entire data domain into different attribute learning areas, and then obtain corresponding learning sub-areas; Step 2: construct an archive node tree for the learning sub-area, wherein the archive node tree includes a root node, a parent node, and a child node; Step 3: Establish relationships between the root node, parent node, and child node of the archive node tree in combination with the hierarchical structure and permutation combination of the node tree; Step 4: Extract features from the data of each learning sub-region and count the number of sub-nodes in all learning sub-regions; Step 5: Introduce a matrix vector, collect the acquired text feature vector, image feature vector and video feature vector, and combine the three feature vectors to form a comprehensive feature vector of the learning sub-region; Step 6: Use a cluster analysis algorithm to obtain identification values, and process and analyze the identification values to obtain a first deep learning set; Step 7: perform entity recognition and extraction on text information, image information, and video information according to the query information and operation records of the archive data, obtain entity segments, and process and analyze the entity segments to obtain a second deep learning set; Step 8: According to the collaborative analysis of the first deep learning set and the second deep learning set, obtain the collaborative analysis deviation, and further process and analyze the collaborative analysis deviation; The process of obtaining target archive data, dividing the entire data domain to obtain different attribute learning areas, and then obtaining corresponding learning sub-areas includes: Obtain target archive data and divide the data into a whole domain; wherein the whole domain includes a first attribute learning area, a second attribute learning area, a third attribute learning area, ..., an mth attribute learning area; m represents the number of divided attribute learning areas; All attribute learning regions are marked as SXQ; various attribute learning regions represent target archive data of different attribute categories, and each attribute category corresponds to a learning region; Each attribute learning region includes at least one learning sub-region ; Subdividing the learning sub-area according to the stored file format; wherein w, p and v represent the text learning sub-area, the image learning sub-area and the video learning sub-area respectively; The process of step 2, constructing an archive node tree for the learning sub-area, includes: Create corresponding root nodes according to different storage formats of learning sub-areas; Creating a number of parent nodes for the root node; Create corresponding child nodes based on several parent nodes; The process of establishing a relationship between the root node, the parent node and the child node of the archive node tree in step 3 by combining the hierarchical structure and permutation combination of the node tree includes: If you learn sub-areas , then the learning sub-region is a text learning sub-region, the text data of the text learning sub-region is taken as the root node, and the text parameters of the text data are recorded as the parent node, and the text type, keywords, index description, page range, date and text size of the text parameters are recorded as child nodes; If you learn sub-areas , then the learning sub-region is an image learning sub-region, the image data of the image learning sub-region is taken as the root node, the image parameters of the image data are recorded as the parent node, and the project type, date and image size of the image parameters are recorded as child nodes; If you learn sub-areas , then the learning sub-area is a video learning sub-area, the video data of the video learning sub-area is taken as the root node, the video parameters of the video data are recorded as the parent node, and the project type, date and video size of the video parameters are recorded as child nodes; The process of step 4, extracting features from the data of each learning sub-region and counting the number of sub-nodes of all learning sub-regions, includes: Extract features from subnodes of different learning sub-areas to obtain text features, image features, and video features; obtain the number of subnodes corresponding to text features. , child node data packet of image features And the subnode data packet of the video feature ; Wherein, d represents the number of the subnode quantity packet of the text feature, and d=1, 2, ..., n; n represents the total number of the subnode quantity packets of the text feature; f represents the number of the subnode data packet of the image feature, and f=1, 2, ..., k; k represents the total number of the subnode data packets of the image feature; g represents the number of the subnode data packet of the video feature, and g=1, 2, ..., j; j represents the total number of the subnode data packets of the video feature; The process of introducing a matrix vector, collecting the acquired text feature vector, image feature vector and video feature vector, and combining the three feature vectors to form a comprehensive feature vector of the learning sub-region includes: Introduce matrix-vector notation based on data feature extraction and child node number statistics: , , , The comprehensive feature vector X of the learning sub-region is calculated using the formula; where, ; In the formula, a is the development coefficient, bi is the support coefficient, D is the support term, r is the support index of the support term D, and R is the preset proportional coefficient of different eigenvectors; i=1, 2, ..., u; and 0<u<1, r is a real number greater than 0; In step 6, a cluster analysis algorithm is used to obtain an identification value, and a process of processing and analyzing the identification value to obtain a first deep learning set includes: The obtained comprehensive feature vector X is mapped into an identification value SY using the clustering analysis algorithm L: SY=L(X); wherein the clustering analysis algorithm selects the K-Means algorithm; By adding the timestamp to the identification value SY, a unique identification value SY0 of the learning sub-area is formed; Obtain a unique identification value set of each learning sub-area, and merge them into a first deep learning set; The step 7, in which the text information, image information and video information are subjected to entity recognition and extraction according to the query information and operation records of the archive data, entity segments are obtained, and the process of processing and analyzing the entity segments to obtain the second deep learning set includes: Extract entity and relationship information from text, images, and video information to obtain single or multiple triples; The extracted entity and relationship information is stored in a learning triple set Trip, where the elements are triples in the form of (c, h, t), where c, h, and t represent text elements, image elements, and video elements, respectively. Count the query times xs and operation frequency zp of archive data in the database; generating entity segments based on data in the set of learned triples; Use the generated entity blocks to construct ternary learning relations to obtain the ternary learning index ; Among them, the expression of the ternary learning index is: ; where ℇ represents the standard variation parameter of the entity segment, are different ternary weight ratio coefficients respectively; Set unique reference values for each entity segment through ternary learning relationship expressions , and merge to obtain a second deep learning set; The step eight, in which the collaborative analysis is performed based on the first deep learning set and the second deep learning set to obtain the collaborative analysis deviation, and the process of further processing and analyzing the collaborative analysis deviation includes: Obtain a first deep learning set H1 and a second deep learning set H2; integrate the first deep learning set H1 and the second deep learning set H2 to calculate a collaborative analysis deviation PL; wherein the formula for the collaborative analysis deviation is: ; In the formula, and Respectively represent the entity segments or learning sub-areas in the first deep learning set and the second deep learning set, and the function It is an indicator or measurement method to measure the relationship between two entity segments. The integral symbol ∫ indicates the integration of all entity segments or learning sub-regions; Setting a critical threshold of the collaborative analysis deviation; comparing each collaborative analysis deviation obtained with the preset critical threshold; Filter out the collaborative analysis deviations that are less than the preset critical threshold, and count the number G of the filtered collaborative analysis deviations; if G is less than or equal to 1, the first deep learning set corresponding to the filtered collaborative analysis deviations is used as the optimal collaborative analysis solution; if G is greater than 1, the first deep learning set is cleared, and a clearing signal is automatically generated, and the first deep learning set is reacquired, and the above operation is repeated.
