A large-scale unstructured data joint processing method and system

Abstract

The application discloses a large-scale unstructured data joint processing method and system, comprising the following steps: S1, collecting large-scale unstructured data and constructing a dynamic metadata feature library containing dynamic updating and index retrieval optimization; S2, constructing a federalized index based on the dynamic metadata feature library, constructing a cross-domain joint index based on the federalized index and performing hierarchical federal architecture optimization to obtain a cross-domain joint optimization index; S3, constructing a global correlation matrix based on the cross-domain joint optimization index to perform cross-domain correlation modeling, and obtaining a global correlation function through the cross-domain correlation modeling; and S4, performing joint optimization solving based on the global correlation function to realize unstructured data joint processing, uniformly optimizing clustering, label prediction and association rule mining and other tasks through fusion processing of a target function, and mutually cooperating between the tasks, so that the whole data processing process is more intelligent and efficient, and the accuracy of data processing results is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for joint processing of large-scale unstructured data. Background Technology

[0002] The amount of data is currently growing explosively, and the data structure is becoming increasingly diversified. Among them, the proportion of unstructured data is increasing day by day. For example, in the medical field, more than 80% of healthcare data is unstructured. Free-format medical texts such as medical records, consultation summaries, and medical records are growing exponentially at the terabyte level every day. In the news industry, news texts, images, and audio, as well as user dynamics and comments in social media, traditional relational databases perform well in processing structured data, but are not effective in dealing with unstructured data.

[0003] In existing unstructured data processing systems, traditional data processing technologies often employ a method of processing different modalities of data separately and then fusing them when dealing with multimodal unstructured data. For example, relational databases based on SMP architecture have weak capabilities in processing unstructured data, while MPP architecture, although improving performance through horizontal scaling, is not good at handling heterogeneous data and is prone to network storms during collaborative computing, limiting cluster size. The lack of a unified fusion mechanism results in complex data processing flows, poor consistency, and difficulty in achieving real-time analysis. Therefore, this paper proposes a method and system for large-scale unstructured data joint processing. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art and to achieve the above objectives, the present invention proposes the following technical solution:

[0005] A method for joint processing of large-scale unstructured data, comprising:

[0006] S1: Collect large-scale unstructured data and build a metadata feature library with dynamic inclusion state, dynamic updates, and optimized index retrieval;

[0007] S2: Build a federated index based on a dynamic metadata feature library, build a cross-domain composite index on the basis of the federated index and optimize the hierarchical federated architecture to obtain a cross-domain composite optimized index;

[0008] S3: Construct a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtain the global association function through cross-domain association modeling;

[0009] S4: Joint optimization based on global correlation function to achieve joint processing of unstructured data.

[0010] The process of building a dynamic metadata feature library is as follows:

[0011] Based on unstructured data, define basic attribute 'a' and content feature 'c', and encapsulate basic attribute 'a' and content feature 'c' into a metadata structure. It uses the distributed document database MongoDB to store metadata collections to form a dynamic metadata feature library, represented as: , where p is the metadata index.

[0012] The index retrieval optimization process is as follows:

[0013] An index is created on the query field of the metadata. For the feature vectors contained in the content feature c, the feature vectors are combined with the vector database Milvus, stored in Milvus, and then fast similarity queries are achieved through cosine similarity. The formula is expressed as: ,in, For two different feature vectors in the metadata set, Representing the eigenvector The length of the mold, Representing the eigenvector The length of the module.

[0014] The process of building a federated index is as follows:

[0015] Distributed indexing technology is used to index the metadata in the metadata feature database across multiple nodes. There are q index nodes, labeled as... For each metadata m, a hash operation is performed based on its content characteristic c, and the metadata m is allocated to an index node using the hash function h(c). j is the federated index. q represents the total number of index nodes, and mod represents the modulo operation.

[0016] The process of obtaining a cross-domain joint optimized index is as follows:

[0017] Define two metadata and The content features are respectively and Calculate the similarity between metadata as a cross-domain composite index;

[0018] Based on cross-domain composite indexes, define the set of participants in the data. ,in, Let i represent the i-th participant, and then construct a global index;

[0019] The cross-domain joint optimization index in hierarchical federated architecture optimization is represented as a relation matrix. And the corresponding global index, i.e. ;

[0020] The global index is built through secure aggregation, expressed by the formula: ,in, This represents the local index corresponding to the participating party. This indicates homomorphic encryption addition, which allows certain types of computations to be performed on encrypted data without first decrypting the data. Indicates an encryption operation;

[0021] The process of obtaining a local index is as follows:

[0022] Using the locality-sensitive hashing algorithm, a family of functions H is defined as follows: Where w is a random vector and b is the offset. For symbolic functions, This represents the probability of similarity preservation. Indicates local data points;

[0023] For a participant Randomly select from function family H A hash function, using the selected one. Each hash function calculates the corresponding local metadata. The hash bucket combination is used to traverse all metadata combinations, grouping metadata with the same hash bucket identifier together to form a local index. .

[0024] The process of constructing the global correlation matrix is ​​as follows:

[0025] Metadata collection for accessing unstructured datasets Each piece of metadata is indexed through the local index of its respective participant. Quickly match similar metadata, extract standardized feature vectors, and simultaneously optimize the global index using cross-domain joint indexing. Construct a global association matrix using the relationship matrix RI. ;

[0026] in, These are the weighting coefficients. Represents two metadata and The original data in the relation matrix RI, This indicates the similarity between two pieces of metadata.

[0027] The process of obtaining the global association function through cross-domain association modeling is as follows:

[0028] Define a global association function. The core objective of the global association function is to achieve data clustering and association rule mining under global association constraints. This is achieved through cross-domain association modeling based on the global association matrix, and is expressed as: ;

[0029] in, It is a data label matrix. It is a cluster center. This indicates the control of clustering loss. The coefficient representing the control of clustering loss This indicates the loss from association rule mining. This represents the coefficient of loss in association rule mining.

[0030] A large-scale unstructured data federated processing system includes:

[0031] Data Acquisition and Construction Module: Collects large-scale unstructured data and constructs a metadata feature library that includes a dynamic update mechanism and index retrieval optimization;

[0032] Index optimization module: Based on the dynamic metadata feature library, a federated index is built. On the basis of the federated index, a cross-domain composite index is built and a hierarchical federated architecture is optimized to obtain a cross-domain composite optimized index.

[0033] Association Acquisition Module: Constructs a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtains the global association function through cross-domain association modeling;

[0034] Processing and optimization module: Based on the global correlation function, joint optimization is performed to achieve joint processing of unstructured data.

[0035] The present invention has the following beneficial effects:

[0036] In this invention, firstly, by constructing a global association matrix, cross-domain semantic association and feature-level similarity are integrated to provide a comprehensive and accurate data association foundation for subsequent tasks such as clustering, label prediction, and association rule mining. For example, in news data processing, it can accurately identify the close association between science and technology news texts and laboratory images, efficiently cluster and integrate related data, and improve data processing efficiency.

[0037] Secondly, by integrating the objective function, tasks such as clustering, label prediction, and association rule mining are optimized in a unified manner, with each task working together. The correlation between data and labels is considered during clustering to optimize label prediction, and the clustering results are referenced during association rule mining, making the entire data processing process more intelligent and efficient, and improving the accuracy of the data processing results.

[0038] Finally, after outputting the results, the cross-domain joint optimization index is updated based on the processing results. Newly discovered relationships are added to the relationship matrix, the hash bucket mapping of the local index is adjusted, and the global index is updated. This mechanism continuously optimizes the index structure, improves the efficiency of the next round of data processing, and identifies the relationships between data more quickly. Attached Figure Description

[0039] Figure 1 This diagram illustrates the steps of a large-scale unstructured data joint processing method and system proposed in this invention.

[0040] Figure 2 This is a system block diagram of a large-scale unstructured data joint processing method and system proposed in this invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1 As shown, the present invention proposes a method for joint processing of large-scale unstructured data, comprising:

[0044] S1: Collect large-scale unstructured data and build a metadata feature library that includes dynamic updates and index retrieval optimization;

[0045] Large-scale unstructured data includes images, audio, sensor data, etc. To acquire raw unstructured data, there are n different data sources, each with a different data format and structure. Preprocessing operations are performed on the collected unstructured data.

[0046] The process of building a dynamic metadata feature library is as follows:

[0047] Based on unstructured data, define basic attribute a and content feature c, where:

[0048] Basic attributes include the data identifier, data size, data format (such as text encoding format, image file format, audio encoding format, etc.), and data source information (such as the name and address of the data source, etc.).

[0049] Content features are extracted based on different data types. For example, for image data, content features include SIFT or HOG features, and for audio data, content features include MFCC audio features.

[0050] The basic attribute 'a' and content feature 'c' are encapsulated into a metadata structure, i.e., m = (a, c), and the metadata set is stored using the distributed document database MongoDB to form a dynamic metadata feature library, represented as: Where p is the metadata index;

[0051] The dynamic update process is as follows:

[0052] After the new data is collected, preprocessed, and metadata is extracted, the MongoDB insertOne method is called to write the metadata document into the collection. If the original data is modified (such as document content being updated), the metadata is extracted again, and an incremental update is performed using updateOne (matching by id) (only updating the c.keywords field).

[0053] Data deletion: When deleting original data, execute deleteOne by ID or mark a.isdeleted:True (soft deletion, easy to trace) to achieve dynamic updates;

[0054] The index retrieval optimization process is as follows:

[0055] An index is created on the query field of the metadata. For the feature vectors contained in the content feature c, the feature vectors are combined with the vector database Milvus, stored in Milvus, and then fast similarity queries are achieved through cosine similarity. The formula is expressed as: ,in, For two different feature vectors in the metadata set, Representing the eigenvector The length of the mold, Representing the eigenvector The modulus length;

[0056] Specifically, by including a metadata feature library with dynamic update mechanisms and optimized index retrieval, it breaks through the inherent pattern of static storage, independent updates and single indexes in traditional metadata management. Through deep collaboration of multi-dimensional feature fusion, linked update logic and adaptive index design, it builds an intelligent management system that is more suitable for large-scale unstructured data environments.

[0057] S2: Build a federated index based on a dynamic metadata feature library, build a cross-domain composite index on the basis of the federated index and optimize the hierarchical federated architecture to obtain a cross-domain composite optimized index;

[0058] The purpose of a federated indexing engine is to integrate and index metadata from different data sources so that it can be queried and retrieved quickly.

[0059] The process of building a federated index is as follows:

[0060] First, distributed indexing technology is used to index the metadata in the metadata feature database across multiple nodes. There are q index nodes, labeled as follows: For each metadata m, a hash operation is performed based on its content characteristic c to determine its position in the index node. The hash function h(c) is then used to allocate the metadata m to the index node. j is the federated index. q represents the total number of index nodes, and mod represents the modulo operation;

[0061] Specifically, by adopting distributed indexing technology, metadata is distributed and stored on multiple index nodes. For each piece of metadata, based on its content feature c, a hash function is used to calculate and the position of m in the index node is determined by modulo operation to build a federated index. This can distribute the processing of large-scale metadata and improve indexing and query efficiency.

[0062] The process of building a cross-domain composite index is as follows:

[0063] Building upon federated indexes, cross-domain composite indexes are established. These indexes consider the semantic relationships between different data sources. For metadata, a similarity calculation method based on feature vectors is used, assuming two text metadata... and ,in, and These are two different identifier indexes and , and ;

[0064] Two text metadata and The content features are respectively and The similarity between metadata is then represented as: , Modulus;

[0065] The similarity between metadata is the cross-domain composite index. This involves retrieving the cross-domain composite indexes of all metadata and filling a relation matrix with the similarity values ​​corresponding to these cross-domain composite indexes. In addition, a hierarchical federated architecture optimization is performed.

[0066] Specifically, for metadata, similarity is calculated based on feature vectors, and the relevance between them is calculated based on the vector dot product and the magnitude. The dot product reflects the consistency of the vector direction, and the magnitude is used for normalization. By calculating the similarity between all metadata, these similarity values ​​are filled into the relationship matrix RI, thereby constructing a more comprehensive index relationship.

[0067] The implementation process of the hierarchical federated architecture optimization is as follows:

[0068] Building upon cross-domain composite indexes, a hierarchical federated architecture is used to further optimize index construction and management, ensuring data security and index efficiency.

[0069] Define the set of participants in the data ,in, Let i represent the i-th participant, and then construct a global index;

[0070] Furthermore, "participants" refers to all parties involved in the entire relationship matrix. In this context, various entities possessing local data and needing to participate in the index building process, such as different subsidiaries in a corporate group's data management scenario, are considered participants. Each party maintains its own local index, which is built based on the data owned by that party.

[0071] The cross-domain joint optimization index in hierarchical federated architecture optimization is represented as a relation matrix. And the corresponding global index, which is constructed through secure aggregation, expressed by the formula:

[0072] ,in, This represents the local index corresponding to the participating party. This indicates homomorphic encryption addition, which allows certain types of computations to be performed on encrypted data without first decrypting the data. Indicates an encryption operation;

[0073] Furthermore, in this scenario, each participant First, index your local index. Encrypt to obtain Then, these encrypted local indices are aggregated using homomorphic encryption addition to obtain the global index. This approach ensures that the data privacy of the participants is protected during the construction of the global index, because the data is stored in encrypted form throughout the process.

[0074] Local Index The acquisition process is as follows:

[0075] The local index is obtained using the locality-sensitive hashing algorithm, and a family of functions H is defined as follows: Where w is a random vector and b is the offset. For symbolic functions, This represents the probability of similarity preservation. Indicates local data points;

[0076] For a participant Its corresponding local metadata is represented as Randomly select from function family H A hash function, using the selected one. Each hash function calculates the corresponding local metadata. The hash bucket combination is used to traverse all metadata combinations, grouping metadata with the same hash bucket identifier together to form a local index. ;

[0077] Furthermore, for any given two metadata... and Similarity Preservation Probability The acquisition process is as follows:

[0078] ,in, Metadata and The probability that the results are the same (i.e., hashed to the same bucket) after being mapped by the same hash function h. Metadata and Similarity between them It is a normalization parameter, and this similarity preservation probability indicates that when any two metadata... and The more similar they are, the higher the probability that they will be hashed into the same bucket;

[0079] Specifically, for the participating parties If two metadata entries have a higher feature similarity, they are more likely to be mapped to the same hash bucket by the same hash function. This characteristic ensures that the local index can efficiently organize similar data, providing a foundation for subsequent cross-domain retrieval and aggregation.

[0080] The final cross-domain joint optimized index is represented as ;

[0081] Specifically, by constructing a cross-domain joint optimized index, efficient and secure management of large-scale unstructured data is achieved. First, by defining participants and local indexes, each participant with local data builds its own local index, which helps manage and protect data privacy. Next, homomorphic encryption is used to securely aggregate the encrypted local indexes of each participant to build a global index. This operation integrates the data index information of all participants while ensuring data privacy. Then, locality-sensitive hashing is used to obtain local indexes. Hash bucket combinations of local metadata are calculated using specific function families and hash functions to efficiently organize similar data, providing a foundation for subsequent retrieval and aggregation. Finally, the relation matrix recording the semantic association of metadata is combined with the global index to form a cross-domain joint optimized index, improving the performance and security of the data index and facilitating cross-domain data querying and retrieval.

[0082] S3: Construct a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtain the global association function through cross-domain association modeling;

[0083] Cross-domain joint optimization index It includes the relation matrix RI and a global index optimized by Locality Sensitive Hash (LSH). First, using the semantic similarity information in the relation matrix RI, preliminary classification or clustering of the data is performed. Let's assume that for any given two metadata... and The similarity between them in the relation matrix can determine whether they belong to the same category;

[0084] The process of constructing the global correlation matrix is ​​as follows:

[0085] Metadata collection for accessing unstructured datasets Each piece of metadata is indexed through the local index of its respective participant. Quickly match similar metadata and extract standardized feature vectors;

[0086] Simultaneously utilize cross-domain joint optimization of the global index. Construct a global association matrix using the relationship matrix RI. ;

[0087] in, These are the weighting coefficients. Represents two metadata and The original data in the relation matrix RI, Indicates the similarity between two metadata items;

[0088] Specifically, local correlation matrix It preserves the cross-domain semantic relationships between different types of data and integrates the similarity at the feature level;

[0089] The process of obtaining the global association function through cross-domain association modeling is as follows:

[0090] Define a global association function. The core objective of the global association function is to achieve data clustering and association rule mining under global association constraints. This is achieved through cross-domain association modeling based on the global association matrix, and is expressed as:

[0091] ;

[0092] in, It is a data label matrix, representing the classification results. It is a cluster center. This indicates the control of clustering loss. The coefficient representing the control of clustering loss This indicates the loss from association rule mining. The coefficient representing the loss from association rule mining;

[0093] Specifically, controlling clustering loss Loss from association rule mining The acquisition process is the process of cross-domain association modeling;

[0094] Furthermore, controlling clustering loss Acquisition process:

[0095] For each data point, calculate its distance to each cluster center, and weight it according to its correlation with the cluster centers. The higher the correlation with the cluster center, the greater the weight of its corresponding distance in the loss function. The formula is as follows:

[0096] ;

[0097] in, Indicates the relationship between the i-th data point and the i-th data point. The correlation between cluster centers is a key indicator; a higher correlation means that the data point is more likely to belong to that cluster center during clustering. It is the i-th local data point. It is the first A vector of cluster centers;

[0098] Specifically, the goal of optimization is to minimize this loss function so that data points are as close as possible to cluster centers with high correlation to them, thereby achieving reasonable clustering.

[0099] Furthermore, the loss from association rule mining. Acquisition process:

[0100] For each pair of data points, the loss of their association rule is calculated based on the degree of association between them. The higher the degree of association between two data points, the greater the weight of their association rule loss in the total loss. The formula is as follows:

[0101] ;

[0102] in, This represents the correlation between the i-th data point and the l-th data point. Given the i-th data point, it represents the conditional probability of the l-th data point occurring.

[0103] Specifically, during the optimization process, minimizing the loss of association rule mining maximizes the conditional probability between data points with high correlation, thereby mining effective association rules.

[0104] Ultimately, by fusing cross-domain semantic associations and feature-level similarity to construct a global association matrix, we can more comprehensively capture the relationships between data. Furthermore, by defining a fusion processing objective function, tasks such as clustering and association rule mining are unified into a single objective function for optimization. This method can simultaneously achieve clustering, label prediction, and association rule mining of unstructured data under global association constraints, improving the efficiency and accuracy of data processing. Then, through iterative optimization and index feedback mechanisms, we can continuously optimize the data processing results and update the index structure, thereby improving the efficiency of the next round of unstructured data processing.

[0105] S4: Joint optimization based on global correlation function to achieve joint processing of unstructured data;

[0106] Based on the gradient descent method, the objective function is iteratively optimized. First, the cluster centers are initialized. The label matrix Y is adjusted according to the global association matrix G in each iteration. (Cluster highly correlated data towards the same center), update Y (make the labels of highly correlated data consistent), calculate conditional probability. (Strengthen the rules for highly correlated data) until the objective function converges, and obtain the final clustering results and association rule set;

[0107] For example, suppose there is a dataset containing a large amount of news data, including unstructured data such as news text, related images, and audio. The system aims to perform clustering, classification, and association rule mining on this data, such as grouping data related to science and technology news into one category and finding the association between news text and related images and audio.

[0108] First, the cluster centers and label matrix are initialized by randomly selecting a portion of news data as the initial cluster centers. For example, 10 news articles can be randomly selected from the dataset as 10 initial cluster centers;

[0109] Then, iterative optimization is performed, based on the adjustment of the global association matrix, and the global association matrix G is calculated. The global association matrix G comprehensively considers the semantic similarity (e.g., the topic similarity of news texts) and feature similarity (e.g., the visual feature similarity of news images) between news data.

[0110] In each iteration, adjust the cluster centers. For each news article, based on its correlation with each cluster center in the global correlation matrix G, the news article is moved to the cluster center with the highest correlation.

[0111] For example, if a technology news article has the highest correlation with a technology news cluster center, then that news article will be moved to that cluster center;

[0112] Update the label matrix Y by updating the labels of the news data according to the correlation between the news data and each label in the global correlation matrix G;

[0113] For example, if a news article is more relevant to the "technology" tag than other tags, then update the tag of this news article to "technology".

[0114] For each pair of news data (e.g., news text and related image), calculate the conditional probability based on their association degree in the global association matrix G. ;

[0115] For example, given a science news article and a related laboratory image, calculate the probability that the laboratory image appears given the news article.

[0116] Repeat the above adjustment and update steps until the global association function converges. The global association function takes into account both clustering loss and association rule loss.

[0117] The final output yields multiple news clusters;

[0118] For example, there are clusters such as "Technology News Cluster", "Sports News Cluster", and "Entertainment News Cluster". Each cluster contains highly relevant news data, and each news article has a clear tag, such as "Technology News Text", "Sports News Image", and "Entertainment News Audio", as well as association rules between news data, such as "There are rules between technology news text and laboratory images".

[0119] Specifically, this association rule indicates that when science and technology news text appears, there will be related laboratory images. This means that in this batch of unstructured datasets, science and technology news reports are often accompanied by laboratory-related images to supplement the explanation.

[0120] Example 2

[0121] like Figure 2 As shown, the present invention proposes a large-scale unstructured data joint processing system, comprising:

[0122] Data Acquisition and Construction Module: Collects large-scale unstructured data and constructs a metadata feature library that includes a dynamic update mechanism and index retrieval optimization;

[0123] Index optimization module: Based on the dynamic metadata feature library, a federated index is built. On the basis of the federated index, a cross-domain composite index is built and a hierarchical federated architecture is optimized to obtain a cross-domain composite optimized index.

[0124] Association Acquisition Module: Constructs a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtains the global association function through cross-domain association modeling;

[0125] Processing and optimization module: Based on the global correlation function, joint optimization is performed to achieve joint processing of unstructured data.

[0126] In the application, several formulas are calculated by removing dimensions and taking their numerical values. The formulas are established by collecting a large amount of data and simulating the most recent real situation. Some coefficients or weights in the formulas are set by those skilled in the art according to the actual situation, so they will not be elaborated here.

[0127] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0128] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for joint processing of large-scale unstructured data, characterized in that, include: S1: Collect large-scale unstructured data and build a metadata feature library that includes dynamic updates and index retrieval optimization; S2: Build a federated index based on a dynamic metadata feature library, build a cross-domain federated index on the basis of the federated index and optimize the hierarchical federated architecture to obtain a cross-domain federated optimized index; The process of obtaining a cross-domain joint optimized index is as follows: Define two metadata and The content features are respectively and Calculate the similarity between metadata as a cross-domain composite index; Based on cross-domain composite indexes, define the set of participants in the data. ,in, Let i represent the i-th participant, and then construct a global index; The cross-domain joint optimization index in hierarchical federated architecture optimization is represented as a relation matrix. And the corresponding global index, i.e. ; The global index is built through secure aggregation, expressed by the formula: ,in, This represents the local index corresponding to the participating party. This indicates homomorphic encryption addition. Indicates an encryption operation; S3: Construct a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtain the global association function through cross-domain association modeling; S4: Joint optimization based on global correlation function to achieve joint processing of unstructured data.

2. The method for joint processing of large-scale unstructured data according to claim 1, characterized in that, The process of building a dynamic metadata feature library is as follows: Based on unstructured data, define basic attribute 'a' and content feature 'c', and encapsulate basic attribute 'a' and content feature 'c' into a metadata structure. It uses the distributed document database MongoDB to store metadata collections to form a dynamic metadata feature library, represented as: , where p is the metadata index.

3. The method for joint processing of large-scale unstructured data according to claim 2, characterized in that, The index retrieval optimization process is as follows: Indexes are created for the query fields of metadata. For the feature vectors contained in content feature c, the feature vectors are combined with the vector database Milvus, stored in Milvus, and then fast similarity queries are achieved using cosine similarity. The formula is expressed as: ,in, For two different feature vectors in the metadata set, Representing the eigenvector The length of the mold, Representing the eigenvector The length of the module.

4. The method for joint processing of large-scale unstructured data according to claim 1, characterized in that, The process of building a federated index is as follows: Distributed indexing technology is used to index the metadata in the metadata feature database across multiple nodes. There are q index nodes, labeled as... For each metadata m, a hash operation is performed based on its content characteristic c, and the metadata m is allocated to an index node using the hash function h(c). j is the federated index. q represents the total number of index nodes, and mod represents the modulo operation.

5. The method for joint processing of large-scale unstructured data according to claim 1, characterized in that, The process of obtaining a local index is as follows: Using the locality-sensitive hashing algorithm, a family of functions H is defined as follows: Where w is a random vector and b is the offset. For symbolic functions, This represents the probability of similarity preservation. Indicates local data points; For a participant Randomly select from function family H A hash function, using the selected one. Each hash function calculates the corresponding local metadata. The hash bucket combination is used to traverse all metadata combinations, grouping metadata with the same hash bucket identifier together to form a local index. .

6. The method for joint processing of large-scale unstructured data according to claim 5, characterized in that, The process of constructing the global correlation matrix is ​​as follows: Metadata collection for accessing unstructured datasets Each piece of metadata is indexed through the local index of its respective participant. Quickly match similar metadata, extract standardized feature vectors, and simultaneously optimize the global index using cross-domain joint indexing. Construct a global association matrix using the relationship matrix RI. ; in, These are the weighting coefficients. Represents two metadata and The original data in the relation matrix RI, This indicates the similarity between two pieces of metadata.

7. The method for joint processing of large-scale unstructured data according to claim 6, characterized in that, The process of obtaining the global association function through cross-domain association modeling is as follows: Define a global association function. The core objective of the global association function is to achieve data clustering and association rule mining under global association constraints. This is achieved through cross-domain association modeling based on the global association matrix, and is expressed as: ; in, It is a data label matrix. It is a cluster center. This indicates the control of clustering loss. The coefficient representing the control of clustering loss This indicates the loss from association rule mining. This represents the coefficient of loss in association rule mining.

8. A large-scale unstructured data joint processing system, implemented according to the method of any one of claims 1-7, characterized in that, include: Data Acquisition and Construction Module: Collects large-scale unstructured data and constructs a metadata feature library that includes a dynamic update mechanism and index retrieval optimization; Index optimization module: Based on the dynamic metadata feature library, a federated index is built. On the basis of the federated index, a cross-domain composite index is built and a hierarchical federated architecture is optimized to obtain a cross-domain composite optimized index. Association Acquisition Module: Constructs a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtains the global association function through cross-domain association modeling; Processing and optimization module: Based on the global correlation function, joint optimization is performed to achieve joint processing of unstructured data.

Images