Large-scale unstructured data joint processing method and system

Abstract

The invention discloses a large-scale unstructured data joint processing method and system. The method comprises the steps that S1, large-scale unstructured data is collected, and a metadata feature library containing dynamic updating and index retrieval optimization is constructed; s2, constructing a federated index based on the dynamic metadata feature library, constructing a cross-domain joint index on the basis of the federated index, and performing layered federated architecture optimization to obtain a cross-domain joint optimization index; s3, constructing a global correlation matrix based on the cross-domain joint optimization index to perform cross-domain correlation modeling, and obtaining a global correlation function through cross-domain correlation modeling; and S4, carrying out joint optimization solution based on the global correlation function to realize unstructured data joint processing, uniformly optimizing tasks such as clustering, label prediction and correlation rule mining by fusing a processing target function, and enabling the tasks to cooperate with each other, so that the whole data processing process is more intelligent and efficient, and the accuracy of a data processing result is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a large-scale unstructured data joint processing method and system. BACKGROUND

[0002] The current data volume is growing explosively, and the data structure is increasingly diversified, among which the proportion of unstructured data is increasing, such as more than 80% of the medical health data in the medical field is unstructured, and the free format medical text such as medical history records, medical records and medical records is growing exponentially at a TB level every day; news text, pictures and audio in the news industry, and user dynamics, comments and other social media, traditional relational databases perform well in processing structured data, but are not good at processing unstructured data. In the existing unstructured data processing system, the traditional data processing technology often adopts the mode of independent module processing different modal data and then fusing when processing multi-modal unstructured data, such as the relational database based on SMP architecture has weak processing ability for unstructured data, MPP architecture improves performance through horizontal expansion, but is not good at processing heterogeneous data, and when cooperative computing, network storm is easy to cause, which limits the size of the cluster, lacks a unified fusion mechanism, resulting in complex data processing flow, poor consistency and difficulty in realizing real-time analysis, so a large-scale unstructured data joint processing method and system are proposed. SUMMARY

[0003] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application proposes the following technical scheme: A large-scale unstructured data joint processing method, comprising: S1: collecting large-scale unstructured data and constructing a dynamic metadata feature library containing dynamic updating and index retrieval optimization; S2: constructing a federated index based on the dynamic metadata feature library, constructing a cross-domain joint index based on the federated index and performing hierarchical federated architecture optimization to obtain a cross-domain joint optimization index; S3: constructing a global correlation matrix based on the cross-domain joint optimization index to perform cross-domain correlation modeling, and obtaining a global correlation function through cross-domain correlation modeling; S4: performing joint optimization solving based on the global correlation function to realize unstructured data joint processing.

[0004] The process of constructing a dynamic metadata feature library is: Based on unstructured data, define basic attributes a and content features c, and encapsulate the basic attributes a and the content features c into a metadata structure , and store the metadata collection in a distributed document database MongoDB to form a dynamic metadata feature library, which is represented as: where p is the metadata index.

[0005] The index retrieval optimization implementation process is: The query field of the metadata is indexed, and the feature vector contained in the content feature c is combined with the vector database Milvus. The feature vector is stored in Milvus, and the cosine similarity is used to realize the fast similarity query, which is expressed by the formula: where is the different two feature vectors in the metadata set, represents the feature vector length, represents the feature vector length.

[0006] The federated index construction process is: Distributed index technology is used to distribute the metadata in the metadata feature library on multiple nodes for indexing. There are q index nodes, marked as For each metadata m, according to its content feature c, the hash operation is performed, and the metadata m is distributed to the index node , j is the federated index, , q is the total number of index nodes, and mod represents the modulo operation.

[0007] The implementation process of obtaining the cross-domain joint optimization index is: The content features of two metadata and are defined as and , and the similarity between the metadata is calculated as the cross-domain joint index; On the basis of the cross-domain joint index, the participant set of the data is defined , where represents the i-th participant, and then the global index is constructed; The performance form of the cross-domain joint optimization index in the hierarchical federated architecture optimization is the relation matrix and the corresponding global index, that is, ; The global index is constructed by secure aggregation, which is expressed by the formula: where represents the local index corresponding to the participant, represents the homomorphic encryption addition, which allows a specific type of calculation on encrypted data without first decrypting the data, represents the encryption operation; The acquisition process of the local index is: A function family H is defined by using the local sensitive hash algorithm, which is represented as: where w is a random vector, b is an offset, is a sign function, denotes a similarity preserving probability, denotes a local data point; for a participant , randomly select hash functions from the function family H, use the selected hash functions to calculate the corresponding hash bucket combination of the local metadata , traverse all metadata combinations, and group the hash buckets that identify the same metadata into a group to form a local index .

[0008] The construction process of the global association matrix is as follows: access the metadata set of the unstructured data set , through the local index of the participant to which each piece of metadata belongs quickly match similar metadata, extract a standardized feature vector, and simultaneously use the global index of the cross-domain joint optimization index and the relationship matrix RI to construct a global association matrix ; wherein is a weight coefficient, denotes the original data of two metadata and in the relationship matrix RI, denotes the similarity between the two metadata.

[0009] The process of obtaining the global association function through cross-domain association modeling is as follows: define a global association function, the core goal of the global association function is to realize data clustering and association rule mining under the global association constraint, which is realized through cross-domain association modeling based on the global association matrix, and is represented as: ; wherein is a data label matrix, is a clustering center, denotes a control clustering loss, denotes a coefficient for controlling the clustering loss, denotes an association rule mining loss, denotes a coefficient for the association rule mining loss.

[0010] A large-scale unstructured data joint processing system comprises: a collection and construction module: collecting large-scale unstructured data and constructing a metadata feature library containing a dynamic update mechanism and index retrieval optimization; Index optimization module: based on the dynamic metadata feature library, a federated index is constructed, and on the basis of the federated index, a cross-domain joint index is constructed and a hierarchical federation architecture optimization is carried out, and a cross-domain joint optimization index is obtained; Correlation acquisition module: based on the cross-domain joint optimization index, a global correlation matrix is constructed to model the cross-domain correlation, and a global correlation function is obtained through cross-domain correlation modeling; Processing optimization module: based on the global correlation function, joint optimization solving is carried out to realize unstructured data joint processing.

[0011] The present application has the following beneficial effects: In the present application, first, by constructing a global correlation matrix, the cross-domain semantic correlation and the similarity at the feature level are fused, providing a comprehensive and accurate data correlation basis for subsequent clustering, label prediction and association rule mining tasks, such as in news data processing, the close correlation between scientific news text and laboratory pictures can be accurately identified, related data can be efficiently clustered and integrated, and the data processing efficiency is improved; Secondly, by fusing the processing target function, the clustering, label prediction and association rule mining tasks are unified and optimized, the tasks are mutually coordinated, the association degree between data and labels is considered during clustering to optimize label prediction, and the clustering results are referred to during association rule mining, so that the entire data processing process is more intelligent and efficient, and the accuracy of the data processing result is improved; Finally, after the result output, the cross-domain joint optimization index is updated according to the processing result, the newly discovered correlation is supplemented to the relationship matrix, the hash bucket mapping of the local index is adjusted, and the global index is updated, which continuously optimizes the index structure, improves the efficiency of the next round of data processing, and more quickly identifies the correlation between data and data. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A method step diagram of a large-scale unstructured data joint processing method and system is provided.

[0013] Figure 2 A system block diagram of a large-scale unstructured data joint processing method and system is provided. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] Embodiment one As Figure 1As shown, the application proposes a large-scale unstructured data joint processing method, which comprises: S1: Collect large-scale unstructured data and construct a dynamic metadata feature library containing dynamic updating and index retrieval optimization; The large-scale unstructured data includes images, audio, sensor data, etc. The original unstructured data is obtained, and there are n different data sources. The data format and structure of each data source are different. The collected unstructured data is preprocessed; The process of constructing a dynamic metadata feature library is as follows: Based on unstructured data, define basic attributes a and content features c, wherein: The basic attributes include the identifier of the data, the size of the data, the format of the data (such as the encoding format of the text, the file format of the image, the encoding format of the audio, etc.), and the data source information (such as the name and address of the data source); The content features are extracted according to different data types, for example, for image data, the content features include SIFT or HOG features, and for audio data, the content features include MFCC audio features; The basic attributes a and content features c are encapsulated into a metadata structure, that is, m= (a, c), and a distributed document database MongoDB is used to store the metadata collection to form a dynamic metadata feature library, which is represented as: Where p is the metadata index; The dynamic updating process is as follows: After the new data is collected, preprocessed, and metadata is extracted, the insertOne method of MongoDB is called to write the metadata document into the collection: if the original data is modified (such as updating the document content), the metadata is extracted again, and the updateOne (matching by id) is used to perform incremental updating (only updating the c.keywords field); Data deletion: when the original data is deleted, deleteOne or mark a.isdeleted:True (soft deletion, convenient for tracing) is executed according to id to realize dynamic updating; The index retrieval optimization implementation process is as follows: Index the query field of the metadata, and for the feature vectors contained in the content features c, store the feature vectors into Milvus, and realize fast similarity query through cosine similarity, which is represented by the formula: Where, is the different 2 feature vectors in the metadata collection, represents the length of the feature vector , represents the length of the feature vector ; Specifically, by including a dynamic updating mechanism and an index retrieval optimized metadata feature library, the inherent mode of static storage, independent updating and single index in traditional metadata management is broken through, and through the deep synergy of multi-dimensional feature fusion, linkage updating logic and adaptive index design, an intelligent management system more suitable for large-scale unstructured data environment is built.

[0016] S2: constructing a federated index based on the dynamic metadata feature library, constructing a cross-domain joint index based on the federated index and performing hierarchical federated architecture optimization to obtain a cross-domain joint optimized index; The purpose of the federated index engine is to integrate and index metadata from different data sources so that it can be quickly queried and retrieved; The federated index construction process is: First, distributed index technology is used to index the metadata in the metadata feature library distributed on multiple nodes, with q index nodes, denoted as For each metadata m, according to its content feature c, a hash operation is performed to determine its position in the index node, and a hash function h(c) is used to distribute the metadata m to the index node , j is the federated index, q is the total number of index nodes, and mod represents the modulo operation; Specifically, by using distributed index technology, the metadata is stored in multiple index nodes, for each metadata, according to its content feature c, a hash function is calculated, and a modulo operation is used to determine the position of m in the index node to construct the federated index, which can disperse large-scale metadata processing and improve index and query efficiency; The construction process of the cross-domain joint index is: On the basis of the federated index, a cross-domain joint index is established, which considers the semantic association between different data sources. For metadata, based on the similarity calculation method of feature vectors, two text metadata and , and are two different identification indexes and , and ; The content features of the two text metadata and are and , respectively, and the similarity between the metadata is represented as: , , represents the modulus; The similarity between the metadata, namely the cross-domain joint index, obtains the cross-domain joint index between all the metadata, and fills the similarity values corresponding to the cross-domain joint indexes into a relation matrix The hierarchical federated architecture optimization is performed in the middle and is layered Specifically, for the metadata, the similarity is calculated based on the feature vectors, and the correlation between them is calculated based on the vector dot product and the module length, wherein the dot product reflects the consistency of the vector direction, and the module length is used for normalization. By calculating the similarity between all the metadata, the similarity values are filled into the relation matrix RI, thereby constructing a more comprehensive index relationship; The implementation process of the hierarchical federated architecture optimization is as follows: On the basis of the cross-domain joint index, the construction and management of the index are further optimized through the hierarchical federated architecture to ensure the security of the data and the efficiency of the index; The set of participants of the data is defined Wherein, represents the i-th participant, and then the global index is constructed; Further, the participants refer to various entities that have local data and need to participate in the index construction process in the entire relation matrix In a data management scenario of an enterprise group, different subsidiaries may be these participants, and each participant maintains its own local index, which is constructed based on the data owned by the participant; The performance form of the cross-domain joint optimization index in the hierarchical federated architecture optimization is the relation matrix and the corresponding global index, which is constructed through secure aggregation, and the formula is as follows: Wherein, represents the local index corresponding to the participant, represents the homomorphic encryption addition, which allows a specific type of calculation to be performed on encrypted data without first decrypting the data, represents the encryption operation; Further, in this scenario, each participant first encrypts its own local index to obtain Then, the encrypted local indexes are aggregated through homomorphic encryption addition to obtain the global index This way ensures that the data privacy of the participants is protected during the construction of the global index, because the data is in an encrypted form throughout the process; The acquisition process of the local index is as follows: The acquisition of local index adopts a local sensitive hashing algorithm, and defines a function family H, which is represented as: wherein w is a random vector, b is an offset, is a symbol function, represents a similarity preserving probability, represents a local data point; For a participant , its corresponding local metadata is represented as , randomly selects hash functions from the function family H, calculates the hash bucket combination of the corresponding local metadata using the selected hash functions, traverses all metadata combinations, groups the metadata with the same hash bucket identification into a group, and forms a local index ; Further, for any given two metadata and , the similarity preserving probability is obtained by the following process: wherein represents the probability that the metadata and are mapped to the same bucket after being mapped by the same hash function h, represents the similarity between the metadata and , and is a normalization parameter. This similarity preserving probability indicates that the probability of two metadata and being hashed into the same bucket is higher when they are more similar; Specifically, for two metadata in the participant , the higher the similarity of their features, the higher the probability of being mapped to the same hash bucket by the same hash function. This feature ensures that the local index can efficiently organize similar data and provide a basis for subsequent cross-domain retrieval and aggregation. The final cross-domain joint optimization index is represented as ; Specifically, by constructing a cross-domain joint optimization index, efficient and secure management of large-scale unstructured data is realized. First, by defining participants and local indexes, each participant with local data constructs a local index according to its own data, which helps to manage and protect data privacy. Then, the encrypted local indexes of each participant are securely aggregated using homomorphic encryption addition to construct a global index, which integrates all participants' data index information while ensuring data privacy. Next, the local sensitive hashing algorithm is used to obtain the local index, and the hash bucket combination of local metadata is calculated through a specific function family and hash function, which efficiently organizes similar data and provides a basis for subsequent retrieval and aggregation. Finally, the relationship matrix of record metadata semantics is combined with the global index to form a cross-domain joint optimization index, which improves the performance and security of data indexing and facilitates cross-domain data query and retrieval.

[0017] S3: Constructing a global correlation matrix based on the cross-domain joint optimization index to perform cross-domain correlation modeling, and obtaining a global correlation function through cross-domain correlation modeling; Cross-domain joint optimization index which contains a relationship matrix RI and a global index optimized by local sensitive hashing (LSH) First, the semantic similarity information in the relationship matrix RI is used to preliminarily classify or cluster the data. For any given two metadata and , their similarity in the relationship matrix can determine whether they belong to the same category. The construction process of the global correlation matrix is as follows: Access the metadata set of the unstructured data set For each metadata, match similar metadata quickly through the local index of its participating party and extract the standardized feature vector; Meanwhile, use the global index of the cross-domain joint optimization index and the relationship matrix RI to construct a global correlation matrix ; where is the weight coefficient, represents the original data of two metadata and in the relationship matrix RI, represents the similarity between the two metadata; Specifically, the global correlation matrix not only retains the cross-domain semantic correlation of different types of data, but also integrates the similarity at the feature level. The process of obtaining the global correlation function through cross-domain correlation modeling is as follows: Define a global association function. The core objective of the global association function is to achieve data clustering and association rule mining under global association constraints. This is achieved through cross-domain association modeling based on the global association matrix, and is expressed as: ; in, It is a data label matrix, representing the classification results. It is a cluster center. This indicates the control of clustering loss. The coefficient representing the control of clustering loss This indicates the loss from association rule mining. The coefficient representing the loss from association rule mining; Specifically, controlling clustering loss Loss from association rule mining The acquisition process is the process of cross-domain association modeling; Furthermore, controlling clustering loss Acquisition process: For each data point, calculate its distance to each cluster center, and weight it according to its correlation with the cluster centers. The higher the correlation with the cluster center, the greater the weight of its corresponding distance in the loss function. The formula is as follows: ; in, Indicates the relationship between the i-th data point and the i-th data point. The correlation between cluster centers is a key indicator; a higher correlation means that the data point is more likely to belong to that cluster center during clustering. It is the i-th local data point. It is the first A vector of cluster centers; Specifically, the goal of optimization is to minimize this loss function so that data points are as close as possible to cluster centers with high correlation to them, thereby achieving reasonable clustering. Furthermore, the loss from association rule mining. Acquisition process: For each pair of data points, the loss of their association rule is calculated based on the degree of association between them. The higher the degree of association between two data points, the greater the weight of their association rule loss in the total loss. The formula is as follows: ; in, This represents the correlation between the i-th data point and the l-th data point. Given the i-th data point, it represents the conditional probability of the l-th data point occurring. Specifically, in the optimization process, the minimization of the association rule mining loss makes the conditional probability between data points with high correlation as high as possible, thereby mining effective association rules; Finally, by fusing the cross-domain semantic association and the feature-level similarity to construct a global association matrix, the relationship between data can be captured more comprehensively, and by defining a fusion processing target function, clustering and association rule mining tasks are unified into one target function for optimization. This method can simultaneously realize clustering, label prediction, and association rule mining of unstructured data under global association constraints, improving the efficiency and accuracy of data processing. Then, through an iterative optimization solution and an index feedback mechanism, the data processing result can be continuously optimized, and the index structure can be updated to improve the efficiency of the next round of unstructured data processing.

[0018] S4: Joint optimization solution based on global association function to realize joint processing of unstructured data; Based on the gradient descent method, the target function is iteratively optimized. First, the clustering centers and the label matrix Y are initialized. In each iteration, the global association matrix G is adjusted (making data with high correlation gather to the same center), Y is updated (making data with high correlation have consistent labels), and the conditional probability (strengthening the rules of highly correlated data) is calculated until the target function converges, obtaining the final clustering result and association rule set; For example, assume there is a dataset containing a large amount of news data, including news text, related pictures, and audio unstructured data. The system goal is to perform clustering classification and association rule mining on these data, such as grouping technology news-related data into one category and finding the association between news text and related pictures and audio; First, the clustering centers and label matrix are initialized. Randomly select a portion of news data as the initial clustering centers , for example, randomly select 10 news articles from the dataset as 10 initial clustering centers; Then, iterative optimization is performed based on the global association matrix adjustment. The global association matrix G is calculated, which considers the semantic similarity (e.g., theme similarity of news text) and feature similarity (e.g., visual feature similarity of news pictures) between news data; In each iteration, the clustering centers are adjusted. For each news data, based on its correlation with each clustering center in the global association matrix G, the news data is moved to the clustering center with the highest correlation; For example, if a technology news has the highest correlation with the technology news clustering center, the news is moved to that clustering center; updating the label matrix Y, updating the label of the news data according to the correlation degree of the news data and each label in the global correlation matrix G; For example, if the correlation degree of a news with the "technology" label is higher than other labels, the label of the news is updated to "technology"; For each pair of news data (such as news text and related pictures), the conditional probability is calculated according to their correlation degree in the global correlation matrix G ; For example, for a technology news text and related laboratory pictures, the probability of the laboratory pictures appearing given the news text is calculated; Repeat the above adjustment and update steps until the global correlation function converges, and the global correlation function considers the clustering loss and the correlation rule loss; The final result is output, and a plurality of news clustering clusters are obtained; For example: "technology news cluster", "sports news cluster", "entertainment news cluster", etc. Each cluster contains news data with high correlation degree, and each news has a clear label, such as "technology news text", "sports news picture", "entertainment news audio", etc. And the correlation rules between news data, such as "there is a rule between technology news text and laboratory pictures"; Specifically, this correlation rule indicates that when technology news text appears, there will be related laboratory pictures, which means that in this batch of unstructured data set, the report of technology news will often be accompanied by laboratory related image data to assist in explanation.

[0019] Embodiment two As shown in Figure 2 The present application provides a large-scale unstructured data joint processing system, which comprises: The acquisition and construction module acquires large-scale unstructured data and constructs a metadata feature library containing a dynamic update mechanism and index retrieval optimization; The index optimization module constructs a federalized index based on the dynamic metadata feature library, constructs a cross-domain joint index based on the federalized index, and performs hierarchical federal architecture optimization to obtain a cross-domain joint optimization index; The correlation acquisition module constructs a global correlation matrix based on the cross-domain joint optimization index to model cross-domain correlation, and obtains a global correlation function through cross-domain correlation modeling; The processing optimization module performs joint optimization solving based on the global correlation function to realize unstructured data joint processing.

[0020] In the application, several formulas involved are calculated by taking the values of the dimensionless formulas, and the establishment of the formulas is obtained by collecting a large amount of data to simulate the most real situation, and part of the coefficients or weights in the formulas is set by the person skilled in the art according to the actual situation, so here is not more elaborated.

[0021] The above embodiments can be implemented totally or partially by software, hardware, firmware or any combination thereof. When implemented by software, the above embodiments can be implemented totally or partially in the form of a computer program product. Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solutions.

[0022] Although the embodiments of the present application have been shown and described, it should be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for joint processing of large-scale unstructured data, characterized in that, include: S1: Collect large-scale unstructured data and build a metadata feature library that includes dynamic updates and index retrieval optimization; S2: Build a federated index based on a dynamic metadata feature library, build a cross-domain federated index on the basis of the federated index and optimize the hierarchical federated architecture to obtain a cross-domain federated optimized index; S3: Construct a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtain the global association function through cross-domain association modeling; S4: Joint optimization based on global correlation function to achieve joint processing of unstructured data.

2. The method for joint processing of large-scale unstructured data according to claim 1, characterized in that, The process of building a dynamic metadata feature library is as follows: Based on unstructured data, define basic attribute 'a' and content feature 'c', and encapsulate basic attribute 'a' and content feature 'c' into a metadata structure. It uses the distributed document database MongoDB to store metadata collections to form a dynamic metadata feature library, represented as: , where p is the metadata index.

3. The method for joint processing of large-scale unstructured data according to claim 2, characterized in that, The index retrieval optimization process is as follows: An index is created on the query field of the metadata. For the feature vectors contained in the content feature c, the feature vectors are combined with the vector database Milvus, stored in Milvus, and then fast similarity queries are achieved through cosine similarity. The formula is expressed as: ,in, For two different feature vectors in the metadata set, Representing the eigenvector The length of the mold, Representing the eigenvector The length of the module.

4. The method for joint processing of large-scale unstructured data according to claim 1, characterized in that, The process of building a federated index is as follows: Distributed indexing technology is used to index the metadata in the metadata feature database across multiple nodes. There are q index nodes, labeled as... For each metadata m, a hash operation is performed based on its content characteristic c, and the metadata m is allocated to an index node using the hash function h(c). j is the federated index. q represents the total number of index nodes, and mod represents the modulo operation.

5. The method for joint processing of large-scale unstructured data according to claim 4, characterized in that, The process of obtaining a cross-domain joint optimized index is as follows: Define two metadata and The content features are respectively and Calculate the similarity between metadata as a cross-domain composite index; Based on cross-domain composite indexes, define the set of participants in the data. ,in, Let i represent the i-th participant, and then construct a global index; The cross-domain joint optimization index in hierarchical federated architecture optimization is represented as a relation matrix. And the corresponding global index, i.e. ; The global index is built through secure aggregation, expressed by the formula: ,in, This represents the local index corresponding to the participating party. This indicates homomorphic encryption addition. This indicates an encryption operation.

6. The method for joint processing of large-scale unstructured data according to claim 5, characterized in that, The process of obtaining a local index is as follows: Using the locality-sensitive hashing algorithm, a family of functions H is defined as follows: Where w is a random vector and b is the offset. For symbolic functions, This represents the probability of similarity preservation. Indicates local data points; For a participant Randomly select from function family H A hash function, using the selected one. Each hash function calculates the corresponding local metadata. The hash bucket combination is used to traverse all metadata combinations, grouping metadata with the same hash bucket identifier together to form a local index. .

7. The method for joint processing of large-scale unstructured data according to claim 6, characterized in that, The process of constructing the global correlation matrix is ​​as follows: Metadata collection for accessing unstructured datasets Each piece of metadata is indexed through the local index of its respective participant. Quickly match similar metadata, extract standardized feature vectors, and simultaneously optimize the global index using cross-domain joint indexing. Construct a global association matrix using the relationship matrix RI. ; in, These are the weighting coefficients. Represents two metadata and The original data in the relation matrix RI, This indicates the similarity between two pieces of metadata.

8. The method for joint processing of large-scale unstructured data according to claim 7, characterized in that, The process of obtaining the global association function through cross-domain association modeling is as follows: Define a global association function. The core objective of the global association function is to achieve data clustering and association rule mining under global association constraints. This is achieved through cross-domain association modeling based on the global association matrix, and is expressed as: ; in, It is a data label matrix. It is a cluster center. This indicates that the clustering loss is controlled. The coefficient representing the control of clustering loss This indicates the loss from association rule mining. This represents the coefficient of loss in association rule mining.

9. A large-scale unstructured data joint processing system, implemented according to the method of any one of claims 1-8, characterized in that, include: Data Acquisition and Construction Module: Collects large-scale unstructured data and constructs a metadata feature library that includes a dynamic update mechanism and index retrieval optimization; Index optimization module: Based on the dynamic metadata feature library, a federated index is built. On the basis of the federated index, a cross-domain composite index is built and a hierarchical federated architecture is optimized to obtain a cross-domain composite optimized index. Association Acquisition Module: Constructs a global association matrix based on cross-domain joint optimization index to perform cross-domain association modeling, and obtains the global association function through cross-domain association modeling; Processing and optimization module: Based on the global correlation function, joint optimization is performed to achieve joint processing of unstructured data.

Images