Archive management method and system based on big data analysis

Abstract

The embodiment of the invention provides an archive management method and system based on big data analysis, and belongs to the technical field of data processing. The method comprises the steps of obtaining initial text data and initial image data corresponding to an initial user; performing deduplication processing on the initial text data to obtain target text data, and performing deduplication processing on the initial image data to obtain target image data; performing association relationship analysis on the target text data and the target image data to obtain a target relationship corresponding to the target text data and the target image data; performing text classification on the target text data to obtain a target type corresponding to the target text data; and performing data storage on the target text data and the target image data according to the target type and the target relationship to obtain a target storage result. The problems that in the prior art, the occupied storage space is increased and the data access and retrieval efficiency is reduced due to the fact that texts and images are often used as independent information units to be processed and an effective integration mechanism is lacked are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a file management method and system based on big data analysis. Background Art

[0002] In the process of archive management, with the rapid development of information and digital technology, data forms are becoming increasingly diversified, among which text data and image data are particularly common. However, this diverse data form also brings new challenges. Especially in the traditional archive management model, text data and image data are often processed as independent information units, lacking an effective integration mechanism. This fragmented data management method not only increases the storage space occupied, but also reduces the efficiency of data access and retrieval. When it is necessary to view the text description and corresponding image of a certain archive at the same time, the user may need to switch back and forth between different systems or databases, which seriously affects work efficiency and user experience. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide an archive management method and system based on big data analysis, aiming to solve the problem that text data and image data in related technologies are often processed as independent information units, lacking an effective integration mechanism, which not only increases the storage space occupied but also reduces the efficiency of data access and retrieval.

[0004] In a first aspect, an embodiment of the present invention provides a file management method based on big data analysis, comprising:

[0005] Obtaining initial archive data, wherein the initial archive data includes initial text data and initial image data corresponding to an initial user;

[0006] Performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data;

[0007] Performing correlation analysis on the target text data and the target image data to obtain a target relationship between the target text data and the target image data;

[0008] Performing text classification on the target text data to obtain a target type corresponding to the target text data;

[0009] The target text data and the target image data are stored according to the target type and the target relationship to obtain a target storage result.

[0010] In a second aspect, an embodiment of the present invention provides an archive management system based on big data analysis, including:

[0011] A data acquisition module, used to obtain initial archive data, wherein the initial archive data includes initial text data and initial image data corresponding to an initial user;

[0012] A data deduplication module, used for performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data;

[0013] A relationship analysis module, used to perform association relationship analysis on the target text data and the target image data to obtain a target relationship between the target text data and the target image data;

[0014] A type determination module, used for performing text classification on the target text data to obtain a target type corresponding to the target text data;

[0015] The data storage module is used to store the target text data and the target image data according to the target type and the target relationship to obtain a target storage result.

[0016] The embodiment of the present invention provides an archive management method and system based on big data analysis, the method comprising: obtaining initial archive data, the initial archive data comprising initial text data and initial image data corresponding to an initial user; performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data; performing correlation analysis on the target text data and the target image data to obtain the target relationship corresponding to the target text data and the target image data; performing text classification on the target text data to obtain the target type corresponding to the target text data; and performing data storage on the target text data and the target image data according to the target type and the target relationship to obtain the target storage result. The method can effectively reduce the storage of duplicate data by performing deduplication processing on the initial text data and the initial image data, thereby improving the data quality and consistency. This helps to ensure the accuracy and uniqueness of the archive data and avoid errors and confusion caused by data redundancy. Then, by performing correlation analysis on the target text data and the target image data, the inherent connection between them can be revealed, thereby enhancing the integration and relevance of the data, reducing the separation between the text data and the image data, and then by classifying the target text data, it can be managed and stored in a targeted manner according to different types, thereby improving the retrievability and organization of the data. Finally, by storing data according to target type and target relationship, the data storage structure can be optimized, making the data more orderly and easier to manage. This not only improves storage efficiency, but also enhances data retrieval and access speed. This solves the problem that text data and image data in related technologies are often processed as independent information units, lacking an effective integration mechanism, which not only increases the storage space occupied, but also reduces the efficiency of data access and retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a process flow of a file management method based on big data analysis provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the module structure of an archive management system based on big data analysis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0022] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] The embodiment of the present invention provides a file management method and system based on big data analysis. The file management method based on big data analysis can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0024] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] Please refer to Figure 1 , Figure 1 A flowchart of a file management method based on big data analysis provided in an embodiment of the present invention.

[0026] like Figure 1 As shown, the archive management method based on big data analysis includes steps S101 to S105.

[0027] Step S101: obtaining initial archive data, wherein the initial archive data includes initial text data and initial image data corresponding to an initial user.

[0028] Exemplarily, the initial archive data corresponding to the initial user is collected from different data sources, and the initial archive data includes initial text data and initial image data corresponding to the initial user. For example, the data source may include electronic documents, scanned images, databases, websites and other forms.

[0029] Step S102: performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data.

[0030] Exemplarily, the initial text data includes first text data and second text data, thereby generating a first hash value corresponding to the first text data and a second hash value corresponding to the second text data, and then comparing the first hash value and the second hash value to determine whether the first text data and the second text data are the same, and then when the first text data and the second text data are the same, retaining either one of them to obtain the target text data.

[0031] Exemplarily, the initial image data includes first image data and second image data, thereby generating a unique first perceptual hash for the first image data and a unique second perceptual hash for the second image data, thereby identifying duplicate images by comparing the first perceptual hash and the second perceptual hash, and then determining the deduplicated image data as the target image data.

[0032] In some embodiments, the initial text data includes at least first text data corresponding to a first source and second text data corresponding to a second source, and the deduplication processing of the initial text data to obtain the target text data includes: performing keyword extraction on the first text data to obtain the corresponding first keyword and performing keyword extraction on the second text data to obtain the second keyword; performing vector representation on the first keyword to obtain a first representation vector and performing vector representation on the second keyword to obtain a second representation vector; calculating a first sentence vector corresponding to the first text data according to a smoothed inverse word frequency algorithm, and fusing the first sentence vector to obtain a second sentence vector corresponding to the first text data; calculating a third sentence vector corresponding to the second text data according to the smoothed inverse word frequency algorithm, and fusing the third sentence vector to obtain a fourth sentence vector corresponding to the second text data; calculating the first representation vector The first distance between the first word vector and the second sentence vector is calculated, and the first keyword is filtered according to the first distance to obtain the third keyword corresponding to the first text data; the second distance between the second representation vector and the fourth sentence vector is calculated, and the second keyword is filtered according to the second distance to obtain the fourth keyword corresponding to the second text data; a text relationship graph corresponding to the first text data and the second text data is constructed according to the third keyword and the fourth keyword, each node in the text relationship graph is the third keyword or the fourth keyword, and each edge is the number of co-occurrences between the two nodes; whether the first text data and the second text data are the same is judged according to the text relationship graph to obtain a target judgment result; and the first text data and the second text data are deduplicated according to the target judgment result to obtain the target text data.

[0033] Exemplarily, a keyword extraction algorithm such as TF-IDF, TextRank, RAKE, etc. is used to extract a number of keywords from the first text data to obtain a first keyword. Similarly, a keyword extraction algorithm is used to extract a number of keywords from the second text data to obtain a second keyword.

[0034] Exemplarily, a word embedding model (such as Word2Vec, GloVe, BERT, etc.) is used to map the first keyword into a high-dimensional space to form a representation vector, thereby obtaining a first representation vector. Similarly, a word embedding model (such as Word2Vec, GloVe, BERT, etc.) is used to map the second keyword into the same high-dimensional space to form a representation vector, thereby obtaining a second representation vector.

[0035] Exemplarily, a smooth inverse word frequency algorithm (such as TF-IDF or BM25) is used to calculate the first weight corresponding to the first keyword, and then a first sentence vector of the associated sentence corresponding to the first keyword in the first text data is obtained by weighted fusion according to the first weight and the first representation vector, and then all the first sentence vectors are fused (such as weighted average, etc.) to obtain the second sentence vector. Similarly, a smooth inverse word frequency algorithm (such as TF-IDF or BM25) is used to calculate the second weight corresponding to the second keyword, and then a third sentence vector of the associated sentence corresponding to the second keyword in the second text data is obtained by weighted fusion according to the second weight and the second representation vector, and then all the third sentence vectors are fused (such as weighted average, etc.) to obtain the fourth sentence vector.

[0036] Exemplarily, the distance between the first characterization vector and the second sentence vector is calculated using the cosine distance calculation formula to obtain a first distance, thereby sorting the first keyword from high to low according to the first distance, thereby retaining keywords with a high correlation with the second sentence vector, and obtaining a third keyword. Similarly, the distance between the second characterization vector and the fourth sentence vector is calculated using the cosine distance calculation formula to obtain a second distance, thereby sorting the second keyword from high to low according to the second distance, thereby retaining keywords with a high correlation with the fourth sentence vector, and obtaining a fourth keyword.

[0037] Exemplarily, the third keyword and the fourth keyword are used as nodes to construct a text relationship graph, and the number of co-occurrences between each node is calculated as the edge between the nodes.

[0038] Exemplarily, a first graph representation corresponding to the first text data under the third keyword and a second graph representation corresponding to the second text data under the fourth keyword are obtained according to the text relationship graph; the similarity between the first graph representation and the second graph representation is calculated, so as to determine whether the first text data and the second text data are the same according to the similarity to obtain a target judgment result. When the similarity is greater than a preset value, the target judgment result between the first text data and the second text data is determined to be the same; when the similarity is less than or equal to the preset value, the target judgment result between the first text data and the second text data is determined to be different.

[0039] Exemplarily, if the target judgment result is the same, one copy of the text data (such as the first text data) is retained, and another copy of the text data (such as the second text data) is deleted to finally obtain the target text data. If the target judgment result is different, both the first text data and the second text data are retained to obtain the target text data.

[0040] In some embodiments, judging whether the first text data and the second text data are identical based on the text relationship graph to obtain a target judgment result includes: calculating a first similarity between each first sentence in the first text data and each node in the text relationship graph, and obtaining a first target sentence in the first text data corresponding to the node in the text relationship graph based on the first similarity; calculating a second similarity between each second sentence in the second text data and each node in the text relationship graph, and obtaining a second target sentence in the second text data corresponding to the node in the text relationship graph based on the second similarity; obtaining a first sentence vector corresponding to the first target sentence in the node and a second sentence vector corresponding to the second target sentence in the node through a twin network model; obtaining a target node vector corresponding to the node based on the first sentence vector and the second sentence vector; determining a target text vector corresponding to the text relationship graph based on the number of co-occurrences between the target node vector and the node; and classifying the target text vector using a text classification model to obtain the target judgment result.

[0041] Exemplarily, for each first sentence in the first text data, the similarity between it and each node in the text relationship graph is calculated to obtain a first similarity. Thus, according to the first similarity, the sentence with the highest similarity between each node and the first sentence in the first text data is selected as the first target sentence of the node in the first text data. Similarly, for each second sentence in the second text data, the similarity between it and each node in the text relationship graph is calculated to obtain a second similarity. Thus, according to the second similarity, the sentence with the highest similarity between each node and the second sentence in the second text data is selected as the second target sentence of the node in the second text data.

[0042] Exemplarily, a twin network model (such as BERT, etc.) is used to vectorize the first target sentence to obtain a first sentence vector. Similarly, the twin network model is used to vectorize the second target sentence to obtain a second sentence vector. Thus, the similarity between any two first sentence vectors and the second sentence vectors is calculated, thereby obtaining the target node vector of each node based on all similarities.

[0043] Exemplarily, the target node vectors are weighted averaged according to the number of co-occurrences between nodes to obtain the target text vector corresponding to the text relationship graph, and then the target text vector is classified using a pre-trained text classification model (such as SVM, deep learning model, etc.) to determine whether the first text data and the second text data are the same, and obtain a target judgment result.

[0044] Specifically, the present application maps the first text data and the second text data to a text relationship graph, thereby judging whether the first text data and the second text data are the same based on the target node vectors corresponding to different nodes in the text relationship graph, thereby more comprehensively and accurately capturing the semantic relationship within the text, thereby improving the accuracy of text similarity judgment. In addition, the introduction of the target node vector makes the nodes in the text relationship graph not only contain the information of the sentence itself, but also covers the association relationship between sentences, thereby enhancing the expressiveness of the text information and further improving the accuracy of the target judgment results.

[0045] Step S103: performing association analysis on the target text data and the target image data to obtain a target relationship corresponding to the target text data and the target image data.

[0046] Exemplarily, a word embedding method (such as Word2Vec, GloVe, etc.) is used to convert the target text data into a vector representation to obtain a first representation result. An image feature extraction method (such as a convolutional neural network CNN) is used to extract a feature vector of the image to obtain a second representation result.

[0047] Exemplarily, the first representation result and the second representation result are aligned to ensure that they are in the same feature space, so as to perform association relationship analysis to obtain a first alignment result corresponding to the first representation result and a second alignment result corresponding to the second representation result.

[0048] Exemplarily, the similarity distance between the first alignment result and the second alignment result is calculated using measurement methods such as cosine similarity and Euclidean distance, so as to obtain the similarity between the target text data and the target image data based on the similarity distance. By setting a suitable threshold, when the similarity is greater than the threshold, the target relationship between the target text data and the target image data is strongly correlated; when the similarity is less than or equal to the threshold, the target relationship between the target text data and the target image data is weakly correlated.

[0049] In some embodiments, the target text data and the target image data are analyzed for association relationships to obtain the target relationship between the target text data and the target image data, including: using the text representation layer of the relationship classification model to perform text feature representation on the target text data to obtain a corresponding text representation vector; using the image representation layer of the relationship classification model to perform image feature representation on the target image data to obtain a corresponding image representation vector; using the text hash layer of the relationship classification model to perform text hash feature representation on the target text data to obtain a corresponding text hash vector; using the image hash layer of the relationship classification model to perform image hash feature representation on the target image data to obtain a corresponding image hash vector; using the text similarity calculation layer of the relationship classification model to perform similarity matrix calculation based on the text representation vector to obtain a corresponding first similarity matrix and perform similarity matrix calculation based on the text hash vector to obtain a corresponding second similarity matrix ; Using the image similarity calculation layer of the relational classification model, a similarity matrix is ​​calculated based on the image representation vector to obtain a corresponding third similarity matrix, and a similarity matrix is ​​calculated based on the image hash vector to obtain a corresponding fourth similarity matrix; Using the modal interaction layer of the relational classification model, a similarity matrix is ​​calculated based on the text representation vector and the image representation vector to obtain a fifth similarity matrix, and a similarity matrix is ​​calculated based on the text hash vector and the image hash vector to obtain a sixth similarity matrix; Using the similarity difference layer of the relational classification model, a feature difference vector between the target text data and the target image data is obtained based on the first similarity matrix, the second similarity matrix, the third similarity matrix, the fourth similarity matrix, the fifth similarity matrix, and the sixth similarity matrix; Using the relation classification layer of the relational classification model, a relationship classification is performed based on the feature difference vector to obtain the target relationship corresponding to the target text data and the target image data.

[0050] Exemplarily, the target text data is input into the text representation layer of the relational classification model, which usually encodes the text using a pre-trained language model (such as BERT, GPT, etc.). The language model processes the text, captures its semantic information, and generates a corresponding text representation vector.

[0051] Exemplarily, the target text data is input into the text hash layer of the relational classification model, which maps the target text data into a binary hash vector through a hash algorithm (such as LSH, SimHash, etc.) to obtain a text hash vector.

[0052] Exemplarily, the target image data is input into the image representation layer of the relational classification model, which usually uses a pre-trained convolutional neural network (such as ResNet, VGG, etc.) to extract the features of the image to obtain an image representation vector. The image representation layer converts the image into a high-dimensional feature vector to capture the visual information of the image.

[0053] Exemplarily, the target image data is input into the image hash layer of the relational classification model, which maps the image into a binary hash vector through a hash algorithm to obtain an image hash vector.

[0054] Exemplarily, the text similarity calculation layer of the relational classification model performs pairwise calculations based on text representation vectors to obtain a first similarity matrix corresponding to the target text data under text features, and performs pairwise calculations based on text hash vectors to obtain a second similarity matrix corresponding to the target text data under hash features.

[0055] Exemplarily, the image similarity calculation layer of the relational classification model performs pairwise calculations based on image representation vectors to obtain a third similarity matrix corresponding to the target image data under image features, and performs pairwise calculations based on image hash vectors to obtain a fourth similarity matrix corresponding to the target image data under hash features.

[0056] Exemplarily, the modal interaction layer of the relational classification model is used to perform pairwise calculations on the text representation vectors and the image representation vectors to obtain the fifth similarity matrix corresponding to the target text data and the target image data under the original features, and the text hash vectors and the image hash vectors are used to perform pairwise calculations to obtain the sixth similarity matrix corresponding to the target text data and the target image data under the hash features.

[0057] Exemplarily, the first matrix difference between the first similarity matrix and the second similarity matrix is ​​calculated using the similarity difference layer, the second matrix difference between the third similarity matrix and the fourth similarity matrix is ​​calculated, the third matrix difference between the fifth similarity matrix and the second similarity matrix is ​​calculated, and the fourth matrix difference between the sixth similarity matrix and the fourth similarity matrix is ​​calculated. Thus, the first matrix difference, the second matrix difference, the third matrix difference and the fourth matrix difference are fused to obtain the characteristic difference vector between the target text data and the target image data. The characteristic difference vector reflects the difference between the target text data and the target image data.

[0058] Exemplarily, the feature difference vector is input to the relationship classification layer, which uses a classifier (such as SVM, deep neural network, etc.) to classify the feature difference vector. The classifier determines the relationship between the target text data and the target image data based on the feature difference vector and outputs the corresponding target relationship. The target relationship includes a strong correlation between the target text data and the target image data, and a weak correlation between the target text data and the target image data.

[0059] Step S104: performing text classification on the target text data to obtain a target type corresponding to the target text data.

[0060] Exemplarily, the target text data is cleaned to remove noise (such as punctuation, special characters, HTML tags, etc.) to ensure the consistency and readability of the text data. Then, a pre-trained language model (such as BERT, GPT, etc.) is used to encode the cleaned target text data to generate a text representation vector containing rich semantic information. Models such as naive Bayes, support vector machines (SVM), random forests, deep neural networks, etc. are used to classify and predict the text representation vectors, and the target type corresponding to the target text data is output. The target type can be other forms of archive types such as character description archives or project description archives.

[0061] In some embodiments, the performing text classification on the target text data to obtain the target type corresponding to the target text data includes: obtaining a type feature corresponding to each initial type in a type set; performing key feature extraction on the target text data to obtain the corresponding target feature; obtaining first data in which the type feature and the target feature appear simultaneously from the target text data and obtaining second data in which the target feature appears but the type feature does not appear; obtaining third data in which the type feature appears but the target feature does not appear from the target text data and obtaining fourth data in which neither the type feature nor the target feature appears; fusing the first data, the second data, the third data and the fourth data to obtain a target feature representation corresponding to the target text data; performing text classification on the target text data according to the target feature representation to obtain the target type corresponding to the target text data; wherein the target feature representation is obtained according to the following formula:

[0062]

[0063] Among them, text i represents the target feature representation corresponding to the i-th target text data, α and β represent the corresponding adjustment parameters, n represents the number of features corresponding to the target feature corresponding to the target text data, m represents the number of texts corresponding to the target text data, and d 1t represents the first data between the tth target feature and the type feature, d 2t represents the second data between the tth target feature and the type feature, d 3t represents the third data between the tth target feature and the type feature, d 4t represents the fourth data between the tth target feature and the type feature, w trepresents the weight information between the t-th target feature and the type feature, and max represents the maximum value.

[0064] Exemplarily, a type set is determined, that is, a set corresponding to the archive type is determined. For each initial type in the type set, the corresponding type feature is extracted. The type feature can be a frequently appearing word, phrase, sentence pattern, etc. in the type, or a language pattern that can represent the characteristics of the type. These features can be extracted by statistical analysis, word frequency analysis, etc.

[0065] Exemplarily, the target text data is preprocessed by cleaning, word segmentation, and stop word removal to ensure the consistency and readability of the target text data. Key features are extracted from the preprocessed target text data. Key features can be important words, phrases, sentence patterns, etc. in the text, or language patterns that can represent the core content of the text.

[0066] Exemplarily, the number of text sentences containing both the type feature and the target feature in the target sentence screened out from the target text data is determined as the first data. The number of text sentences containing only the target feature but not the type feature in the target sentence screened out from the target text data is determined as the second data. The number of text sentences containing only the type feature but not the target feature in the target sentence screened out from the target text data is determined as the third data. The number of text sentences containing neither the type feature nor the target feature in the target sentence screened out from the target text data is determined as the fourth data.

[0067] Exemplarily, the first data, the second data, the third data and the fourth data are fused according to the following formula to obtain the target feature representation of the target text data:

[0068]

[0069] Among them, text i represents the target feature representation corresponding to the i-th target text data, α and β represent the corresponding adjustment parameters, n represents the number of features corresponding to the target feature corresponding to the target text data, m represents the number of texts corresponding to the target text data, and d 1t Represents the first data between the tth target feature and the type feature, d 2t Represents the second data between the tth target feature and the type feature, d 3t Represents the third data between the tth target feature and the type feature, d 4t represents the fourth data between the tth target feature and the type feature, w t Represents the weight information between the tth target feature and the type feature, and max indicates the maximum value.

[0070] Exemplarily, by fusing the first data, the second data, the third data, and the fourth data, the matching degree between the target features and the type features can be comprehensively extracted from different angles, thereby generating a more representative and discriminative target feature representation by integrating multi-angle data information, flexibly adjusting parameters, and weighted fusion, thereby providing support for subsequent target type identification.

[0071] Exemplarily, the initial type corresponding to when the target feature representation is maximum is determined as the target type corresponding to the target text data.

[0072] Step S105: storing the target text data and the target image data according to the target type and the target relationship to obtain a target storage result.

[0073] Exemplarily, when the target relationship is strongly correlated, mutual reference parameters are added between the target text data and the target image data, so that data of another modality can be obtained through the mutual reference parameters in subsequent queries; when the target relationship is weakly correlated, there is no need to add mutual reference parameters between the target text data and the target image data.

[0074] Exemplarily, when the target relationship is strongly correlated, the target text data and the mutual reference parameters between the target text data and the target image data are stored in the database corresponding to the target type to obtain the target storage result; when the target relationship is weakly correlated, the target text data and the target image data are respectively stored in the database corresponding to the target type to obtain the target storage result.

[0075] In some implementations, the method further includes: obtaining target query content corresponding to the target user; performing a data query in the target storage result according to the target query content to obtain a target query result corresponding to the target user.

[0076] For example, an intuitive user interface, such as a web form, mobile application interface, or command line interface, is designed for the target user to enter the query content. Ensure that the interface is concise and easy to operate, and then guide the target user to enter the corresponding target query content on the interface. For example, keywords, problem descriptions, or other relevant information can be entered in the input box.

[0077] Exemplarily, a corresponding query statement is constructed based on the target query content input by the target user. The query statement needs to match the structure and data type of the storage result to ensure that the required data can be accurately queried. The constructed query statement is executed in the target storage result to perform a data query operation. The query can be executed by manually operating a database management tool, a file search tool, or an API call. The target query result is obtained and saved for subsequent processing and display.

[0078] In some embodiments, performing data query in the target storage result according to the target query content to obtain the target query result corresponding to the target user includes: performing text classification on the target query content to obtain a query type, and obtaining associated query content corresponding to the query type from the target storage result; performing text feature representation on the target query content according to a text processing layer of a text similarity model to obtain a first text vector and performing text feature representation on associated text content in the associated query content to obtain a second text vector; performing similarity calculation on the first text vector and the second text vector according to a vector calculation layer of the text similarity model to obtain a similarity calculation result; performing text interaction calculation on the text similarity model according to the text similarity model The layer uses the similarity calculation result in combination with the second text vector to obtain the first target feature of the target query content relative to the associated text content; the text interaction layer according to the text similarity model uses the similarity calculation result in combination with the first text vector to obtain the second target feature of the associated text content relative to the target query content; according to the feature fusion layer of the text similarity model, the first target feature and the second target feature are information-fused to obtain a target fusion feature; according to the similarity prediction layer of the text similarity model, a similarity value is predicted for the target fusion feature to obtain a target similarity value corresponding to the associated text content; and the associated query content is sorted according to the target similarity value to obtain the target query result.

[0079] Exemplarily, a pre-trained text classification model or a rule-based classification method is used to perform text classification on the target query content to obtain a query type, and the query type may be a corporate profile query or a personal profile query, and the like.

[0080] Exemplarily, according to the query type, the associated query content related to the query type is retrieved in the target storage result. For example, if it is a corporate archive query, the database corresponding to the corporate archive is determined as the associated query content.

[0081] Exemplarily, the text processing layer in the text similarity model, such as word embedding, is used to characterize the text features of the target query content to obtain a first text vector, and the associated text content in the associated query content is processed in the same manner to obtain a second text vector.

[0082] Exemplarily, the vector calculation layer in the text similarity model is used to perform similarity calculation on the first text vector and the second text vector using methods such as cosine similarity and Euclidean distance to obtain a similarity calculation result.

[0083] Exemplarily, the text interaction layer in the text similarity model is used to combine the similarity calculation result and the second text vector to generate a first target feature of the target query content relative to the associated text content.

[0084] For example, the first text vector is composed of vector 1, vector 2 and vector 3, while the second text vector is composed of vector 4, vector 5 and vector 6. In the similarity calculation result, the distance between any vector in the first text vector and any vector in the second text vector is recorded, and then the similarity between any two vectors in the first text vector is calculated, for example, similarity 1 between vector 1 and vector 2 is calculated, and similarity 2 between vector 1 and vector 3 is calculated, and then the similarity between vector 1 and any vector in the second text vector in the similarity calculation result is summed to obtain the similarity sum value, and then similarity 1 is divided by the similarity sum value and similarity 2 is divided by the similarity sum value to sum and multiply with the second text vector to obtain the first feature corresponding to vector 1, and similarly obtain the second feature corresponding to vector 2 and the third feature corresponding to vector 3, and finally vector 1, vector 2 and vector 3 are average-pooled to obtain the first target feature of the target query content relative to the associated text content.

[0085] Exemplarily, the text interaction layer of the text similarity model utilizes the similarity calculation result in combination with the first text vector to obtain the second target feature of the associated text content relative to the target query content.

[0086] For example, the first text vector is composed of vector 1, vector 2 and vector 3, while the second text vector is composed of vector 4, vector 5 and vector 6. In the similarity calculation result, the distance between any vector in the first text vector and any vector in the second text vector is recorded, and then the similarity between any two vectors in the second text vector is calculated, for example, the similarity 4 between vector 4 and vector 5 is calculated, and the similarity 5 between vector 4 and vector 6 is calculated, and then the similarity between vector 4 and any vector in the first text vector in the similarity calculation result is summed to obtain the similarity sum value, and then the similarity 4 is divided by the similarity sum value, and the similarity 5 is divided by the similarity sum value to sum and multiply with the first text vector to obtain the fourth feature corresponding to vector 4, and similarly obtain the fifth feature corresponding to vector 5 and the sixth feature corresponding to vector 6, and finally, vector 4, vector 5 and vector 6 are average-pooled to obtain the second target feature of the associated text content relative to the target query content.

[0087] Exemplarily, the feature fusion layer in the text similarity model is used to fuse the first target feature and the second target feature. For example, the first target feature and the second target feature are multiplied to obtain a product vector and the first target feature and the second target feature are subtracted to obtain a difference vector, and then the difference vector and the product vector are concatenated to obtain the target fusion feature.

[0088] Exemplarily, the similarity prediction layer in the text similarity model uses a mapping function to predict the similarity value of the target fusion feature to obtain the target similarity value corresponding to the associated text content.

[0089] Exemplarily, all associated query contents are sorted in descending order according to the target similarity value, that is, the associated query contents with the highest target similarity value are placed at the front. Several associated query contents with the highest ranking are selected as the final target query result. Usually, the top 10 or more ranked contents can be selected, and the specific number can be adjusted according to actual needs and the relevance of the query results.

[0090] In some embodiments, after obtaining the target query result corresponding to the target user, the method further includes: obtaining the target permissions corresponding to the target user, and filtering the target query results according to the target permissions to obtain the final query results corresponding to the target user; visualizing the query results according to the final query results to obtain the target visualization results corresponding to the target user.

[0091] Exemplarily, obtain the permission information corresponding to the target user. The permission information may include the user's role, access level, viewable data range, etc. Filter the previously obtained target query results according to the permission information of the target user. Ensure that only data that meets the user's permissions is retained. For example, if the target user can only view data of a specific department, filter out the data of other departments, and save the filtered query results as the final query results.

[0092] For example, according to the characteristics of the final query results and user needs, a suitable visualization chart is designed. For example, different types of charts such as bar charts, line charts, pie charts, maps, etc. can be used to display data. Thus, the visualization tool is used to generate the target visualization result.

[0093] Through the above steps, we can obtain the target user's permissions, filter the query results, and visualize the final query results to generate the target visualization results corresponding to the target user. This process not only ensures that users can only view data that meets their permissions, but also improves the understanding and use efficiency of data through visualization, providing strong support for decision-making.

[0094] In some embodiments, visualizing the query results based on the final query results to obtain the target visualization results corresponding to the target user includes: performing fuzzy clustering on the final query results to obtain the target clustering results; obtaining the cluster center corresponding to each cluster in the target clustering results; visualizing the query results based on the cluster center to obtain the target visualization results corresponding to the target user.

[0095] Exemplarily, a fuzzy clustering method, such as fuzzy C-means (FCM) clustering, is selected to perform fuzzy clustering on the final query results, and clustering parameters (such as the number of clusters, fuzzy factors, etc.) are configured according to the data characteristics and clustering requirements of the final query results, and then the clustering algorithm is run to generate the target clustering results.

[0096] Exemplarily, the cluster center corresponding to each cluster is obtained according to the target clustering result, so that the cluster center of each cluster is used as the first branch, and the remaining content in each cluster is used as the second branch under the first branch, so that the target user can select more interesting content or content that better meets the needs according to the cluster center after viewing the cluster center, and then continue to view other content under the cluster center, thereby reducing the time for users to summarize and screen by themselves.

[0097] For each cluster, calculate its corresponding cluster center. The cluster center can help the target user quickly understand the main archival information of each cluster. Take the cluster center of each cluster as the first branch. Each cluster center represents an independent category or theme, and the target user can quickly understand the distribution and main characteristics of the overall data by viewing these cluster center points. Under each cluster center, list the remaining contents in the cluster. These contents are specific data points or query results distributed around the cluster center. In this way, the target user can quickly locate the category of interest based on the cluster center, and view more detailed information under the category, and then use appropriate visualization tools or software, such as Tableau, Power BI, D3.js, etc., to visualize the above hierarchical structure. These tools usually provide a wealth of chart types and interactive functions, which can better display the hierarchical structure of the data, so as to obtain the target visualization results.

[0098] Through the above steps, after viewing the cluster center, the target user can select the content that he is more interested in or more in line with his needs based on the cluster center, and then continue to view other content under the cluster center. This method not only reduces the time for users to summarize and filter by themselves, but also improves the readability and ease of use of query results, providing users with a more intuitive and efficient way to obtain information.

[0099] See also Figure 2 , Figure 2An archive management system 200 based on big data analysis is provided in an embodiment of the present application. The archive management system 200 based on big data analysis includes a data acquisition module 201, a data deduplication module 202, a relationship analysis module 203, a type determination module 204, and a data storage module 205, wherein the data acquisition module 201 is used to obtain initial archive data, and the initial archive data includes initial text data and initial image data corresponding to an initial user; the data deduplication module 202 is used to perform deduplication processing on the initial text data to obtain target text data and to perform deduplication processing on the initial image data to obtain target image data; the relationship analysis module 203 is used to perform association relationship analysis on the target text data and the target image data to obtain the target relationship corresponding to the target text data and the target image data; the type determination module 204 is used to perform text classification on the target text data to obtain the target type corresponding to the target text data; the data storage module 205 is used to perform data storage on the target text data and the target image data according to the target type and the target relationship to obtain a target storage result.

[0100] In some implementations, the initial text data includes at least first text data corresponding to a first source and second text data corresponding to a second source. In the process of performing deduplication processing on the initial text data to obtain target text data, the data deduplication module 202 performs:

[0101] Performing keyword extraction on the first text data to obtain a corresponding first keyword and performing keyword extraction on the second text data to obtain a second keyword;

[0102] Performing vector representation on the first keyword to obtain a first representation vector and performing vector representation on the second keyword to obtain a second representation vector;

[0103] Calculating a first sentence vector corresponding to the first text data according to a smoothed inverse word frequency algorithm, and fusing the first sentence vectors to obtain a second sentence vector corresponding to the first text data;

[0104] Calculating a third sentence vector corresponding to the second text data according to the smoothed inverse word frequency algorithm, and fusing the third sentence vector to obtain a fourth sentence vector corresponding to the second text data;

[0105] Calculating a first distance between the first representation vector and the second sentence vector, and filtering the first keyword according to the first distance to obtain a third keyword corresponding to the first text data;

[0106] Calculating a second distance between the second representation vector and the fourth sentence vector, and filtering the second keyword according to the second distance to obtain a fourth keyword corresponding to the second text data;

[0107] Constructing a text relationship graph corresponding to the first text data and the second text data according to the third keyword and the fourth keyword, wherein each node in the text relationship graph is the third keyword or the fourth keyword, and each edge is the number of co-occurrences between two nodes;

[0108] Judging whether the first text data and the second text data are the same according to the text relationship graph, and obtaining a target judgment result;

[0109] The first text data and the second text data are deduplicated according to the target judgment result to obtain the target text data.

[0110] In some implementations, the data deduplication module 202, in the process of determining whether the first text data and the second text data are identical according to the text relationship graph and obtaining a target determination result, executes:

[0111] Calculating a first similarity between each first sentence in the first text data and each node in the text relationship graph, and obtaining a first target sentence in the first text data corresponding to the node in the text relationship graph according to the first similarity;

[0112] Calculating a second similarity between each second sentence in the second text data and each node in the text relationship graph, and obtaining a second target sentence in the second text data corresponding to the node in the text relationship graph according to the second similarity;

[0113] Obtaining a first sentence vector corresponding to the first target sentence in the node and a second sentence vector corresponding to the second target sentence in the node through a twin network model;

[0114] Obtain a target node vector corresponding to the node according to the first sentence vector and the second sentence vector;

[0115] Determine a target text vector corresponding to the text relationship graph according to the target node vector and the number of co-occurrences between the nodes;

[0116] The target text vector is classified using a text classification model to obtain the target judgment result.

[0117] In some implementations, the relationship analysis module 203 performs, during the process of performing association relationship analysis on the target text data and the target image data to obtain the target relationship corresponding to the target text data and the target image data:

[0118] Using the text representation layer of the relational classification model to perform text feature representation on the target text data to obtain a corresponding text representation vector;

[0119] Using the image representation layer of the relational classification model to represent the target image data with image features to obtain a corresponding image representation vector;

[0120] Using the text hash layer of the relational classification model to perform text hash feature representation on the target text data to obtain a corresponding text hash vector;

[0121] Using the image hash layer of the relational classification model to perform image hash feature representation on the target image data to obtain a corresponding image hash vector;

[0122] Using the text similarity calculation layer of the relational classification model to perform similarity matrix calculation according to the text representation vector to obtain a corresponding first similarity matrix and to perform similarity matrix calculation according to the text hash vector to obtain a corresponding second similarity matrix;

[0123] Using the image similarity calculation layer of the relational classification model, performing similarity matrix calculation according to the image representation vector to obtain a corresponding third similarity matrix, and performing similarity matrix calculation according to the image hash vector to obtain a corresponding fourth similarity matrix;

[0124] Using the modal interaction layer of the relational classification model, a similarity matrix is ​​calculated based on the text representation vector and the image representation vector to obtain a fifth similarity matrix, and a similarity matrix is ​​calculated based on the text hash vector and the image hash vector to obtain a sixth similarity matrix;

[0125] Obtaining a feature difference vector between the target text data and the target image data according to the first similarity matrix, the second similarity matrix, the third similarity matrix, the fourth similarity matrix, the fifth similarity matrix, and the sixth similarity matrix using the similarity difference layer of the relational classification model;

[0126] The target relationship corresponding to the target text data and the target image data is obtained by performing relationship classification according to the feature difference vector using the relationship classification layer of the relationship classification model.

[0127] In some implementations, the type determination module 204, in the process of performing text classification on the target text data to obtain the target type corresponding to the target text data, performs:

[0128] Get the type characteristics corresponding to each initial type in the type set;

[0129] Extract key features from the target text data to obtain corresponding target features;

[0130] Obtaining from the target text data first data in which the type feature and the target feature appear at the same time, and obtaining second data in which the target feature appears but the type feature does not appear;

[0131] Obtaining from the target text data third data in which the type feature appears but the target feature does not appear, and obtaining fourth data in which neither the type feature nor the target feature appears;

[0132] fusing the first data, the second data, the third data and the fourth data to obtain a target feature representation corresponding to the target text data;

[0133] Performing text classification on the target text data according to the target feature representation to obtain the target type corresponding to the target text data;

[0134] The target feature representation is obtained according to the following formula:

[0135]

[0136] Among them, text i represents the target feature representation corresponding to the i-th target text data, α and β represent the corresponding adjustment parameters, n represents the number of features corresponding to the target feature corresponding to the target text data, m represents the number of texts corresponding to the target text data, and d 1t represents the first data between the tth target feature and the type feature, d 2t represents the second data between the tth target feature and the type feature, d 3t represents the third data between the tth target feature and the type feature, d 4t represents the fourth data between the tth target feature and the type feature, w t represents the weight information between the t-th target feature and the type feature, and max represents the maximum value.

[0137] In some implementations, the archive management system 200 based on big data analysis further performs:

[0138] Obtain target query content corresponding to target users;

[0139] A data query is performed in the target storage result according to the target query content to obtain a target query result corresponding to the target user.

[0140] In some implementations, the archive management system 200 based on big data analysis performs, in the process of performing a data query in the target storage result according to the target query content and obtaining the target query result corresponding to the target user:

[0141] Performing text classification on the target query content to obtain a query type, and obtaining associated query content corresponding to the query type from the target storage result;

[0142] Performing text feature representation on the target query content according to the text processing layer of the text similarity model to obtain a first text vector and performing text feature representation on the associated text content in the associated query content to obtain a second text vector;

[0143] Performing similarity calculation on the first text vector and the second text vector according to the vector calculation layer of the text similarity model to obtain a similarity calculation result;

[0144] Obtaining, according to the text interaction layer of the text similarity model, a first target feature of the target query content relative to the associated text content by combining the similarity calculation result with the second text vector;

[0145] According to the text interaction layer of the text similarity model, the similarity calculation result is combined with the first text vector to obtain a second target feature of the associated text content relative to the target query content;

[0146] According to the feature fusion layer of the text similarity model, information fusion is performed on the first target feature and the second target feature to obtain a target fusion feature;

[0147] Performing similarity value prediction on the target fusion feature according to the similarity prediction layer of the text similarity model to obtain a target similarity value corresponding to the associated text content;

[0148] The associated query contents are sorted according to the target similarity values ​​to obtain the target query results.

[0149] In some implementations, after obtaining the target query result corresponding to the target user, the archive management system 200 based on big data analysis further performs:

[0150] Obtaining the target authority corresponding to the target user, and filtering the target query result according to the target authority to obtain the final query result corresponding to the target user;

[0151] The query result is visualized according to the final query result to obtain a target visualization result corresponding to the target user.

[0152] In some implementations, the archive management system 200 based on big data analysis, in the process of visualizing the query result according to the final query result and obtaining the target visualization result corresponding to the target user, executes:

[0153] Performing fuzzy clustering on the final query result to obtain a target clustering result;

[0154] Obtaining the cluster center corresponding to each cluster in the target clustering result;

[0155] The query result is visualized according to the cluster center to obtain the target visualization result corresponding to the target user.

[0156] In some implementations, the archive management system 200 based on big data analysis may be applied to a terminal device.

[0157] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the archive management system 200 based on big data analysis described above can refer to the corresponding process in the aforementioned archive management method embodiment based on big data analysis, and will not be repeated here.

[0158] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of the archive management method based on big data analysis provided in the description of the embodiment of the present invention.

[0159] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0160] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0161] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0162] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A file management method based on big data analysis, characterized in that: The method comprises: Obtaining initial archive data, wherein the initial archive data includes initial text data and initial image data corresponding to an initial user; Performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data; Performing correlation analysis on the target text data and the target image data to obtain a target relationship between the target text data and the target image data; Performing text classification on the target text data to obtain a target type corresponding to the target text data; The target text data and the target image data are stored according to the target type and the target relationship to obtain a target storage result.

2. The method according to claim 1, characterized in that The initial text data at least includes first text data corresponding to a first source and second text data corresponding to a second source, and performing deduplication processing on the initial text data to obtain target text data includes: Performing keyword extraction on the first text data to obtain a corresponding first keyword and performing keyword extraction on the second text data to obtain a second keyword; Performing vector representation on the first keyword to obtain a first representation vector and performing vector representation on the second keyword to obtain a second representation vector; Calculating a first sentence vector corresponding to the first text data according to a smoothed inverse word frequency algorithm, and fusing the first sentence vectors to obtain a second sentence vector corresponding to the first text data; Calculating a third sentence vector corresponding to the second text data according to the smoothed inverse word frequency algorithm, and fusing the third sentence vector to obtain a fourth sentence vector corresponding to the second text data; Calculating a first distance between the first representation vector and the second sentence vector, and filtering the first keyword according to the first distance to obtain a third keyword corresponding to the first text data; Calculating a second distance between the second representation vector and the fourth sentence vector, and filtering the second keyword according to the second distance to obtain a fourth keyword corresponding to the second text data; Constructing a text relationship graph corresponding to the first text data and the second text data according to the third keyword and the fourth keyword, wherein each node in the text relationship graph is the third keyword or the fourth keyword, and each edge is the number of co-occurrences between two nodes; Judging whether the first text data and the second text data are the same according to the text relationship graph, and obtaining a target judgment result; The first text data and the second text data are deduplicated according to the target judgment result to obtain the target text data.

3. The method according to claim 2, characterized in that The step of judging whether the first text data and the second text data are identical according to the text relationship graph to obtain a target judgment result includes: Calculating a first similarity between each first sentence in the first text data and each node in the text relationship graph, and obtaining a first target sentence in the first text data corresponding to the node in the text relationship graph according to the first similarity; Calculating a second similarity between each second sentence in the second text data and each node in the text relationship graph, and obtaining a second target sentence in the second text data corresponding to the node in the text relationship graph according to the second similarity; Obtaining a first sentence vector corresponding to the first target sentence in the node and a second sentence vector corresponding to the second target sentence in the node through a twin network model; Obtain a target node vector corresponding to the node according to the first sentence vector and the second sentence vector; Determine a target text vector corresponding to the text relationship graph according to the target node vector and the number of co-occurrences between the nodes; The target text vector is classified using a text classification model to obtain the target judgment result.

4. The method according to claim 1, characterized in that: The performing association analysis on the target text data and the target image data to obtain the target relationship corresponding to the target text data and the target image data includes: Using the text representation layer of the relational classification model to perform text feature representation on the target text data to obtain a corresponding text representation vector; Using the image representation layer of the relational classification model to represent the target image data with image features to obtain a corresponding image representation vector; Using the text hash layer of the relational classification model to perform text hash feature representation on the target text data to obtain a corresponding text hash vector; Using the image hash layer of the relational classification model to perform image hash feature representation on the target image data to obtain a corresponding image hash vector; Using the text similarity calculation layer of the relational classification model to perform similarity matrix calculation according to the text representation vector to obtain a corresponding first similarity matrix and to perform similarity matrix calculation according to the text hash vector to obtain a corresponding second similarity matrix; Using the image similarity calculation layer of the relational classification model, performing similarity matrix calculation according to the image representation vector to obtain a corresponding third similarity matrix, and performing similarity matrix calculation according to the image hash vector to obtain a corresponding fourth similarity matrix; Using the modal interaction layer of the relational classification model, a similarity matrix is ​​calculated based on the text representation vector and the image representation vector to obtain a fifth similarity matrix, and a similarity matrix is ​​calculated based on the text hash vector and the image hash vector to obtain a sixth similarity matrix; Obtaining a feature difference vector between the target text data and the target image data according to the first similarity matrix, the second similarity matrix, the third similarity matrix, the fourth similarity matrix, the fifth similarity matrix, and the sixth similarity matrix using the similarity difference layer of the relational classification model; The target relationship corresponding to the target text data and the target image data is obtained by performing relationship classification according to the feature difference vector using the relationship classification layer of the relationship classification model.

5. The method according to claim 1, characterized in that The performing text classification on the target text data to obtain a target type corresponding to the target text data includes: Get the type characteristics corresponding to each initial type in the type set; Extract key features from the target text data to obtain corresponding target features; Obtaining from the target text data first data in which the type feature and the target feature appear at the same time, and obtaining second data in which the target feature appears but the type feature does not appear; Obtaining from the target text data third data in which the type feature appears but the target feature does not appear, and obtaining fourth data in which neither the type feature nor the target feature appears; fusing the first data, the second data, the third data and the fourth data to obtain a target feature representation corresponding to the target text data; Performing text classification on the target text data according to the target feature representation to obtain the target type corresponding to the target text data; The target feature representation is obtained according to the following formula: Among them, text i represents the target feature representation corresponding to the i-th target text data, α and β represent the corresponding adjustment parameters, n represents the number of features corresponding to the target feature corresponding to the target text data, m represents the number of texts corresponding to the target text data, and d 1t represents the first data between the tth target feature and the type feature, d 2t represents the second data between the tth target feature and the type feature, d 3t represents the third data between the tth target feature and the type feature, d 4t represents the fourth data between the tth target feature and the type feature, w t represents the weight information between the t-th target feature and the type feature, and max represents the maximum value.

6. The method according to claim 1, characterized in that The method further comprises: Obtain target query content corresponding to target users; A data query is performed in the target storage result according to the target query content to obtain a target query result corresponding to the target user.

7. The method according to claim 6, characterized in that The performing of data query in the target storage result according to the target query content to obtain the target query result corresponding to the target user includes: Performing text classification on the target query content to obtain a query type, and obtaining associated query content corresponding to the query type from the target storage result; Performing text feature representation on the target query content according to the text processing layer of the text similarity model to obtain a first text vector and performing text feature representation on the associated text content in the associated query content to obtain a second text vector; Performing similarity calculation on the first text vector and the second text vector according to the vector calculation layer of the text similarity model to obtain a similarity calculation result; Obtaining, according to the text interaction layer of the text similarity model, a first target feature of the target query content relative to the associated text content by combining the similarity calculation result with the second text vector; According to the text interaction layer of the text similarity model, the similarity calculation result is combined with the first text vector to obtain a second target feature of the associated text content relative to the target query content; According to the feature fusion layer of the text similarity model, information fusion is performed on the first target feature and the second target feature to obtain a target fusion feature; Performing similarity value prediction on the target fusion feature according to the similarity prediction layer of the text similarity model to obtain a target similarity value corresponding to the associated text content; The associated query contents are sorted according to the target similarity values ​​to obtain the target query results.

8. The method according to claim 6, characterized in that After obtaining the target query result corresponding to the target user, the method further includes: Obtaining the target authority corresponding to the target user, and filtering the target query result according to the target authority to obtain the final query result corresponding to the target user; The query result is visualized according to the final query result to obtain a target visualization result corresponding to the target user.

9. The method according to claim 8, characterized in that The visualizing the query result according to the final query result to obtain the target visualization result corresponding to the target user includes: Performing fuzzy clustering on the final query result to obtain a target clustering result; Obtaining the cluster center corresponding to each cluster in the target clustering result; The query result is visualized according to the cluster center to obtain the target visualization result corresponding to the target user.

10. A file management system based on big data analysis, characterized in that: include: A data acquisition module, used to obtain initial archive data, wherein the initial archive data includes initial text data and initial image data corresponding to an initial user; A data deduplication module, used for performing deduplication processing on the initial text data to obtain target text data and performing deduplication processing on the initial image data to obtain target image data; A relationship analysis module, used to perform association relationship analysis on the target text data and the target image data to obtain a target relationship between the target text data and the target image data; A type determination module, used for performing text classification on the target text data to obtain a target type corresponding to the target text data; The data storage module is used to store the target text data and the target image data according to the target type and the target relationship to obtain a target storage result.

Images