A multi-source heterogeneous big data processing system

Abstract

The present invention relates to the field of data processing technology. The present invention discloses a multi-source heterogeneous big data processing system, which includes dividing data into structured sub-data and unstructured sub-data, and performing feature extraction on the unstructured sub-data to obtain multimodal features, so that the data can be compared in a unified feature space. Then, pattern matching is used to identify the similarity and correlation between the data, ensuring that the data of the same entity are directly fused to reduce redundancy and improve accuracy. Then, an optimization algorithm is used to select the most similar data combination, and the actual relationship type between the data is determined through relationship deduction. Finally, the most appropriate fusion method is selected based on these relationships. The present invention ensures the accuracy and effectiveness of the data fusion process, and provides a reliable basis for more in-depth analysis and decision-making.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a multi-source heterogeneous big data processing system. Background Art

[0002] With the rapid development of information technology, the amount of data held by governments at all levels and large enterprises is growing at an exponential rate. However, the diversity of data sources leads to differences in formats, the uneven data quality affects the application effect, the decentralized data management weakens the relevance and integration of data, and the heterogeneity of data forms brings semantic differences and communication barriers. In this context, the task of multi-source heterogeneous data fusion is particularly important. It aims to effectively integrate multimodal data from different sources. At present, most existing research focuses on multimodal deep learning, while there is relatively little research on how to fuse multi-source heterogeneous government data.

[0003] In the existing technology, the Chinese patent with publication number CN114817208A provides a method for constructing a multi-source heterogeneous government data organization representation and fusion management model. The patent classifies, cleans, identifies and encrypts government data, and uses metadata parsing and correlation analysis to build a complete data model to achieve efficient storage and management. The Chinese patent with publication number CN116719864A provides a method for integrating multi-source heterogeneous data in the government cloud. The patent uses the standard deviation method to perform noise reduction preprocessing on government data, and based on data similarity classification and heterogeneous data association information mining model, improves data quality and fusion efficiency.

[0004] Although existing technologies have integrated multi-source heterogeneous government data, there are often different types of relationships between entities in government data, such as identity relationships and stage relationships. It is difficult for existing technologies to determine appropriate integration methods based on these relationships.

[0005] In view of this, the present invention proposes a multi-source heterogeneous big data processing system to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a multi-source heterogeneous big data processing system.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] In a first aspect, a multi-source heterogeneous big data processing system includes:

[0009] Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features;

[0010] Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R;

[0011] Relationship deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination, obtain deduction results, and determine the target fusion method based on the deduction results. The deduction results at least include stage relationships, and the target fusion method at least includes stage fusion. One element represents one intermediate sub-data.

[0012] Furthermore, the method of extracting features from unstructured sub-data includes:

[0013] Encode G elements in the unstructured sub-data to obtain corresponding feature vectors, calculate the semantic similarity between each feature vector, and fuse the feature vectors whose semantic similarity is greater than a preset first similarity threshold to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0014] Furthermore, the method of fusing feature vectors includes:

[0015] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0016] Furthermore, the method for pattern matching based on structured sub-data and multimodal features includes:

[0017] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, and the second entity information corresponding to each feature vector in the multimodal feature is obtained. The structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0018] Furthermore, the method of dividing the data to be processed according to data characteristics includes:

[0019] The data features are input into the pre-built structural classification model to obtain the data classification results, which include structural sub-data and unstructured sub-data.

[0020] The construction methods of structural classification models include:

[0021] Obtain Q groups of training data, where Q is a positive integer greater than 1. The training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model.

[0022] Furthermore, the optimization algorithm is a particle swarm optimization algorithm. The method for determining the optimal sub-data combination based on the R intermediate sub-data and the particle swarm optimization algorithm includes:

[0023] The first initial unit is used to define the particle's position vector as:

[0024] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw ], each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data.

[0025] The second initial unit is used to define the fitness function as:

[0026]

[0027] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, and Sim(·) is the similarity function.

[0028] Velocity update unit: used to update the particle's velocity based on the particle's current velocity, individual optimal position, and global optimal position.

[0029] Position update unit: used to update the position of the particle according to the new velocity:

[0030] x i (t+1)=x i (t)+vi (t+1);

[0031] Where x i (t+1) is the updated position of the particle at time t+1, x i (t) is the position of the particle at time t.

[0032] Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position p is updated. best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best .

[0033] Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

[0034] Furthermore, the method for updating particle velocity includes:

[0035] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0036] Where, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position in the entire particle swarm, p best is the current historical optimal position of particle i.

[0037] Furthermore, the method of deducing the relationship between each element in the optimal sub-data combination includes:

[0038] Inputting the optimal sub-data combination into the relational deduction model, thereby obtaining the deduction result output by the relational deduction model;

[0039] The methods for constructing the relationship deduction model include:

[0040] A sample data set is obtained, where the sample data set includes a historical optimal sub-data combination and historical deduction results. The sample data set is divided into a sample training set and a sample test set. A regression network is constructed, where the historical optimal sub-data combination in the sample training set is used as input data of the regression network, and the historical deduction results in the sample training set are used as output data of the regression network. The regression network is trained to obtain an initial regression network for predicting real-time deduction results. The initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention divides data into structured sub-data and unstructured sub-data, and extracts features from the unstructured sub-data to obtain multimodal features, so that the data can be compared in a unified feature space. Then, pattern matching is used to identify the similarities and correlations between the data, ensuring that the data of the same entity are directly fused to reduce redundancy and improve accuracy. Next, an optimization algorithm is used to select the most similar data combination, and relationship deduction is used to determine the actual relationship type between the data. Finally, the most appropriate fusion method is selected based on these relationships. In this way, the present invention ensures the accuracy and effectiveness of the data fusion process and provides a reliable basis for more in-depth analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the structure of a multi-source heterogeneous big data processing system in the present invention;

[0044] Figure 2 This is a flow chart of a multi-source heterogeneous big data processing method in the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] Example 1

[0047] See also Figure 1 As shown, this embodiment discloses a multi-source heterogeneous big data processing system, including:

[0048] Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features;

[0049] In this embodiment, the data to be processed refers to multi-source heterogeneous data sent by government platforms at all levels or large enterprises. These data may include structured data (such as database records, spreadsheets), unstructured data (such as text files, emails, social media content) and multimodal data (such as images, audio, video). These data come from diverse sources and in different formats, and usually involve multiple aspects such as government management, public services, and business operations.

[0050] Among them, the data characteristics include at least structural measurement values ​​and label density values. The structural measurement values ​​represent the proportion of the data to be processed that conforms to a specific format (such as tabular form, field name, and data type consistency), and the label density value represents the ratio of the number of labels in the data to be processed. It can be understood that structured data usually appears in a fixed tabular form, such as rows and columns, field names and data types (such as SQL databases, spreadsheets), while unstructured data lacks such a fixed format (such as text files, images, audio, and video). Therefore, structured data conforms to a specific format. Structured data usually contains labels and attributes to describe the hierarchical relationship of data elements, while unstructured data usually does not contain such label information. Therefore, the larger the structural measurement value and label density value, the greater the probability that the data to be processed is structured data.

[0051] Methods for dividing the data to be processed according to data characteristics include:

[0052] The data features are input into the pre-built structural classification model to obtain the data classification results, which include structural sub-data and unstructured sub-data.

[0053] The construction methods of structural classification models include:

[0054] Obtain Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model. The classifier is preferably a naive Bayes model or a support vector machine model.

[0055] Methods for extracting features from unstructured sub-data include:

[0056] Encode G elements in the unstructured sub-data to obtain corresponding feature vectors, calculate the semantic similarity between each feature vector, and fuse the feature vectors whose semantic similarity is greater than a preset first similarity threshold to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0057] It should be noted that the unstructured sub-data includes G elements, each of which can be text data, image data or video data. When the element is text data, a word embedding model (such as Word2Vec, GloVe) is used to convert the text data into a feature vector. When the element is image data, a convolutional neural network (such as ResNet, Inception) or a visual Transformer (ViT) is used to encode the image and extract the feature vector representing the image content. When the element is video data, the video data can be divided into several image data for processing, or a 3D convolutional neural network (3D-CNN) can be used to encode the video data. The semantic similarity mentioned above represents the semantic similarity or correlation between different elements (text, image, video). Specifically, the semantic similarity measures whether these different types of data are similar or related in content, theme, concept or semantics.

[0058] It should be added that the method for calculating semantic similarity can be implemented by calculating Euclidean distance or cosine similarity, and the first similarity threshold is pre-set based on expert experience.

[0059] Methods for fusing feature vectors include:

[0060] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0061] It can be understood that when the semantic similarity is greater than the preset second similarity threshold, it indicates that the two feature vectors are in a high similarity situation, the data features are highly similar in semantics and information, and feature splicing can be performed directly to maintain the integrity and relevance of the information. When the semantic similarity is greater than the preset first similarity threshold, it indicates that the two feature vectors are in a medium similarity situation, the data features have a certain degree of complementarity, and the contribution of each feature in the fusion process can be dynamically adjusted through the attention mechanism.

[0062] Among them, feature splicing and fusion is the most direct and commonly used feature-level fusion method, which directly splices feature vectors from different modalities (such as text, image, audio, video, etc.) into a larger feature vector.

[0063] Attention mechanism fusion uses the attention mechanism to dynamically assign weights to feature vectors of different modalities to determine the importance of each feature in the fusion process. This method can flexibly select and focus on the most relevant information and filter out unimportant or noisy features.

[0064] For example, when performing attention fusion on the feature vector corresponding to the text and the feature vector corresponding to the image, it is necessary to first input the feature vector corresponding to the text and the feature vector corresponding to the image into the attention fusion formula. The attention fusion formula is specifically:

[0065]

[0066] In the formula, Attention(·) represents the attention mechanism, TFV represents the feature vector corresponding to the text, IFV represents the feature vector corresponding to the image, VK is the value of the feature vector corresponding to the image, Softmax(·) is the probability distribution function, FV, IFV T is the transposed dot product between the feature vector corresponding to the text and the feature vector corresponding to the image, d IFV is the dimension of the feature vector corresponding to the image.

[0067] It should be noted that there is a difference between IFV and VK mentioned above. For example, if there is a picture of an illegal building, then IFV can be the shape and material information of the building's roof, and VK can be the height of the illegal building or the texture information of the illegal building's roof. T The representation is to calculate the similarity between FV and IFV. The above probability distribution function can convert the calculated similarity into a weight distribution, which is used to weight the eigenvalues ​​of the image. The final Attention (TFV, IFV, VK) is a feature vector that integrates the semantic relationship between text and image. This weighted feature vector can more effectively capture the association information between text and image.

[0068] In this embodiment, due to the diverse types of unstructured data (such as text, images, and videos), directly extracting features from data of different modalities may result in inconsistent features and difficulty in comparison. By encoding each element into a feature vector, different types of data can be mapped to the same feature space, so that these data can be compared with each other. Directly extracting features of all unstructured sub-data may result in excessive information, including a large amount of irrelevant or redundant information. By calculating the semantic similarity between feature vectors and only fusing those features whose semantic similarity is higher than a threshold, irrelevant features can be filtered out, unnecessary noise can be reduced, and the effectiveness of data fusion can be improved.

[0069] Furthermore, this embodiment does not simply splice all similarity features, which may introduce a large amount of irrelevant or redundant information, resulting in increased noise in the fusion result and reducing the validity of the data. By setting two different similarity thresholds, it is possible to more accurately control which features should be directly spliced ​​and which require an attention mechanism to judge their relative importance, thereby ensuring the accuracy and relevance of the fusion results and improving the quality of the fused data.

[0070] Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R;

[0071] In this embodiment, pattern matching refers to the use of multimodal features of structured data and unstructured data to identify, compare and associate the same or similar data patterns or entities during the data fusion process. The goal of pattern matching is to find similarities or correlations between data by using structured sub-data (such as database records, spreadsheets, etc.) and multimodal features extracted from unstructured sub-data (such as text, images, videos), and then determine which data belong to the same entity or the same subject or concept. For example, in government data processing, heterogeneous data from multiple sources may be involved, and all data related to an event (such as a text report on environmental pollution, a satellite image of a pollution source, and a video surveillance record) must be matched. Through pattern matching, different modal data describing the same event can be combined together to help more comprehensive understanding and decision-making.

[0072] Methods for pattern matching based on structured sub-data and multimodal features include:

[0073] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, and the second entity information corresponding to each feature vector in the multimodal feature is obtained. The structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0074] It should be noted that entity information refers to the characteristics or identifiers in the data that represent specific things, objects, events or concepts. These entity information are used to identify the specific content described by the data. For example, in a government data fusion task for an environmental pollution incident, government agencies need to comprehensively analyze data from different sources in order to have a more comprehensive understanding of the pollution incident and take corresponding measures.

[0075] In this scenario, structured data (structured sub-data) may come from environmental pollution report records in the database, such as pollution source information, pollutant type, occurrence time, pollution degree, etc.; while unstructured data (unstructured sub-data) may include a news report about the pollution incident, or an image showing the geographical location of the pollution source factory A and its surrounding conditions.

[0076] By vectorizing the structured sub-data using One-Hot encoding or word embedding technology, a feature vector representing the data can be generated. At the same time, the fused multimodal features can be a feature representation that combines the text embedding vector and the image feature vector. Since the first entity information and the second entity information both point to the same entity - "factory pollution", the environmental pollution report records, news reports and images showing the pollution source factory A can be clustered through clustering methods, thereby combining related data together to facilitate subsequent data fusion and analysis.

[0077] It should be added that the clustering method mentioned above may be through K-means clustering or DBSCAN, etc., which will not be described in detail in this embodiment.

[0078] In this embodiment, by calculating the semantic similarity of the features of unstructured data and fusing them into multimodal features, and then clustering and matching them with structured data, this combination strategy can more accurately determine the similarity and correlation between data. Multimodal feature fusion helps to capture richer information, and the clustering process ensures that only highly related data are clustered together, thereby improving the accuracy and reliability of the data fusion results. In addition, in the unstructured data feature extraction stage, similar feature vectors are merged by calculating semantic similarity, which effectively reduces redundant data. Subsequently, when pattern matching and clustering are performed on structured data and multimodal features, the processing efficiency is greatly improved because the data volume has been optimized and compressed. Such a combination method can effectively reduce computing costs and improve the overall efficiency of large-scale data processing and fusion.

[0079] It should be noted that, in this embodiment, the data to be fused of the same entity are directly fused to form R intermediate sub-data. The intermediate sub-data refers to the comprehensive vector after the fusion of the structure sub-vector and the feature vector. That is to say, the data to be fused contains the structure sub-vector and the feature vector. Only when the first entity information of the structure sub-vector matches the second entity information of the feature vector, can these data to be fused be fused.

[0080] Since pattern matching not only clusters the same entities but also clusters similar entities, and this embodiment requires subsequent relationship deduction, it is necessary to ensure that the data belongs to the same entity. Only under the premise of the same entity can the accuracy and consistency of data fusion be ensured, providing a reliable foundation for subsequent relationship deduction.

[0081] In this context, the data to be fused for the same entity can be directly fused through feature concatenation or weighted average fusion. If weighted average fusion is chosen, it is necessary to pre-assign a corresponding weight to each vector, for example, a weight of 0.6 for the structure subvector and a weight of 0.4 for the feature vector. This approach can balance the contributions of different types of features to the fusion results, more accurately reflecting the information of the same entity and providing higher-quality data support for subsequent analysis and deduction.

[0082] Relationship deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain a deduction result, and determine the target fusion method based on the deduction result, wherein the deduction result at least includes a stage relationship, the target fusion method at least includes stage fusion, and each element represents one intermediate sub-data;

[0083] In this embodiment, the optimal sub-data combination refers to the combination of the most similar intermediate sub-data. For example, suppose there is a river pollution incident that spans the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties. Therefore, the most similar intermediate sub-data combination must be generated based on these intermediate sub-data to support further analysis and decision-making.

[0084] The optimization algorithm is the particle swarm optimization algorithm. The method of determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm optimization algorithm includes:

[0085] The first initial unit is used to define the particle's position vector as:

[0086] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw], each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data;

[0087] The particle swarm optimization algorithm gradually approaches the optimal solution by simulating the movement of particles in a search space. The particle positions represent a vectorized representation of candidate solutions. This embodiment initializes the particle positions to establish the algorithm's search space, enabling the algorithm to begin evaluating and optimizing different sub-data combinations.

[0088] The second initial unit is used to define the fitness function as:

[0089]

[0090] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, Sim(·) is the similarity function;

[0091] Among them, the fitness function is used as the algorithm optimization target to evaluate the quality of each particle (sub-data combination). The goal is to maximize the similarity of sub-data within the combination. Through the fitness function, the particles are guided to move towards the optimal solution in the search space to ensure that the similarity in the sub-data combination is maximized.

[0092] Speed ​​update unit: used to update the speed of particles according to the current speed, individual optimal position and global optimal position of particles:

[0093] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0094] Where, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position in the entire particle swarm, p best is the current historical optimal position of particle i;

[0095] Among them, the particle swarm algorithm simulates the movement of particles in the search space by updating the speed, prompting particles to move towards the historical best position and the global best position. Through speed updating, it balances the relationship between exploring new areas and developing existing optimal areas, ensuring that particles can effectively explore the entire search space and avoid falling into local optimal solutions.

[0096] Position update unit: used to update the position of the particle according to the new velocity:

[0097] x i (t+1)=x i (t)+v i (t+1);

[0098] Where x i (t+1) is the updated position at time t+1, x i (t) is the position of the particle at time t;

[0099] The new position is determined by the current position and velocity, simulating the movement of particles. Updating the position enables particles to move in the search space, thereby approaching or exploring new possible solutions.

[0100] Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position p is updated. best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best ;

[0101] This embodiment needs to ensure that each particle continuously learns and updates its position to find the possible optimal solution; the update of the global optimal position ensures that the search direction of the entire particle swarm has a common optimization goal.

[0102] Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

[0103] This embodiment constructs a fitness function based on the similarity of sub-data combinations and intermediate sub-data, and can accurately evaluate the internal similarity of each combination. In this way, the algorithm can maximize the aggregation of similar sub-data, ensure that the fused data represents the same entity or subject, and avoid the risk of data confusion or misfusion. The selection of the optimal sub-data combination can effectively integrate intermediate sub-data from different sources but with the same or similar semantics by maximizing similarity to form a more comprehensive and coherent data set. This integration provides a richer perspective and a more complete data foundation for subsequent data analysis, relationship deduction and decision-making. Since the fusion of data is based on similarity optimization, it ensures that the data in the combination has similar attributes and characteristics, which provides a more solid foundation for subsequent relationship deduction. Deduction and analysis on such a high-similarity data set can more accurately identify complex relationships between entities and support more in-depth scenario analysis and strategic decision-making.

[0104] Methods for deducing the relationship between each element in the optimal sub-data combination include:

[0105] The optimal sub-data combination is input into the relational deduction model to obtain the deduction result output by the relational deduction model.

[0106] The methods for constructing the relationship deduction model include:

[0107] A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and historical deduction results, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, and the historical deduction results in the sample training set are used as output data of the regression network, the regression network is trained, and an initial regression network for predicting real-time deduction results is obtained, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model, where the regression network is a deep neural network model.

[0108] It can be understood that relationship deduction refers to analyzing sub-data combinations to find out the logical relationship, temporal relationship or causal relationship between them. For example, suppose there is a river pollution incident that spans the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties to generate corresponding sub-data combinations. At this time, the deduction result is a stage relationship, because when analyzing river pollution incidents, the occurrence, spread and impact of pollution is often a dynamic process, which will go through multiple different development stages. The deduction results can help understand the evolution process and timeline of pollution incidents by revealing the relationship between these stages, thereby better guiding response and decision-making.

[0109] It should be added that, in addition to stage relationships, there are also causal relationships, identical relationships, and hierarchical relationships. For example, in a pollution incident analysis, there may be monitoring and response data at different levels such as national, provincial, municipal, and county levels. There is a hierarchical relationship between these data, which reflects the division of responsibilities and information flow between management agencies at all levels. This embodiment does not go into too much detail on this.

[0110] It can be understood that after determining the deduction result in this embodiment, the corresponding target fusion method is determined according to the corresponding relationship. For example, when the deduction result is a stage relationship, the target fusion method is stage fusion. Stage fusion refers to the merging and integration of data in stages according to the different stages to which the data belongs during the data fusion process. By fusing the data of each stage separately, and then gradually integrating these fusion results according to the time or event development sequence, a continuous data reflecting the overall evolution process of the event can be obtained. It should be noted that stage fusion is only a fusion method. The specific fusion technical means can be voting fusion or weighted average fusion, etc. The purpose is to comprehensively and accurately reflect the dynamic change process of the event and provide a logically clear, coherent and orderly data basis for further analysis and decision-making.

[0111] This embodiment divides the data into structured sub-data and unstructured sub-data, and performs feature extraction on the unstructured sub-data to obtain multimodal features, so that the data can be compared in a unified feature space. Then, pattern matching is used to identify the similarity and correlation between the data, ensuring that the data of the same entity is directly fused to reduce redundancy and improve accuracy. Then, an optimization algorithm is used to select the most similar data combination, and relationship deduction is used to determine the actual relationship type between the data. Finally, the most appropriate fusion method is selected based on these relationships. In this way, this embodiment ensures the accuracy and effectiveness of the data fusion process and provides a reliable foundation for more in-depth analysis and decision-making.

[0112] Example 2

[0113] See also Figure 2 As shown, based on the same inventive concept, this embodiment discloses a method for processing multi-source heterogeneous big data. For details not provided in this embodiment, please refer to the description of the relevant parts in Example 1. The method includes:

[0114] S10: Obtaining data features corresponding to the data to be processed, dividing the data to be processed into structured sub-data and unstructured sub-data according to the data features, performing feature extraction on the unstructured sub-data to obtain multimodal features;

[0115] In this embodiment, the data to be processed refers to multi-source heterogeneous data sent by government platforms at all levels or large enterprises. These data may include structured data (such as database records, spreadsheets), unstructured data (such as text files, emails, social media content) and multimodal data (such as images, audio, video). These data come from diverse sources and in different formats, and usually involve multiple aspects such as government management, public services, and business operations.

[0116] Methods for dividing the data to be processed according to data characteristics include:

[0117] The data features are input into the pre-built structural classification model to obtain the data classification results, which include structural sub-data and unstructured sub-data.

[0118] The construction methods of structural classification models include:

[0119] Obtain Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model. The classifier is preferably a naive Bayes model or a support vector machine model.

[0120] Methods for extracting features from unstructured sub-data include:

[0121] Encode G elements in the unstructured sub-data to obtain corresponding feature vectors, calculate the semantic similarity between each feature vector, and fuse the feature vectors whose semantic similarity is greater than a preset first similarity threshold to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0122] Methods for fusing feature vectors include:

[0123] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0124] Among them, feature splicing and fusion is the most direct and commonly used feature-level fusion method, which directly splices feature vectors from different modalities (such as text, image, audio, video, etc.) into a larger feature vector.

[0125] Attention mechanism fusion uses the attention mechanism to dynamically assign weights to feature vectors of different modalities to determine the importance of each feature in the fusion process. This method can flexibly select and focus on the most relevant information and filter out unimportant or noisy features.

[0126] For example, when performing attention fusion on the feature vector corresponding to the text and the feature vector corresponding to the image, it is necessary to first input the feature vector corresponding to the text and the feature vector corresponding to the image into the attention fusion formula. The attention fusion formula is specifically:

[0127]

[0128] In the formula, Attention(·) represents the attention mechanism, TFV represents the feature vector corresponding to the text, IFV represents the feature vector corresponding to the image, VK is the value of the feature vector corresponding to the image, Softmax(·) is the probability distribution function, FV, IFVT is the transposed dot product between the feature vector corresponding to the text and the feature vector corresponding to the image, d IFV is the dimension of the feature vector corresponding to the image.

[0129] S20: Perform pattern matching based on the structural sub-data and multimodal features to determine N data to be fused, directly fuse the data to be fused of the same entity to obtain R intermediate sub-data, N ≥ R;

[0130] In this embodiment, pattern matching refers to the use of multimodal features of structured data and unstructured data to identify, compare and associate the same or similar data patterns or entities during the data fusion process. The goal of pattern matching is to find similarities or correlations between data by using structured sub-data (such as database records, spreadsheets, etc.) and multimodal features extracted from unstructured sub-data (such as text, images, videos), and then determine which data belong to the same entity or the same subject or concept. For example, in government data processing, heterogeneous data from multiple sources may be involved, and all data related to an event (such as a text report on environmental pollution, a satellite image of a pollution source, and a video surveillance record) must be matched. Through pattern matching, different modal data describing the same event can be combined together to help more comprehensive understanding and decision-making.

[0131] Methods for pattern matching based on structured sub-data and multimodal features include:

[0132] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, and the second entity information corresponding to each feature vector in the multimodal feature is obtained. The structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0133] It should be noted that entity information refers to the characteristics or identifiers in the data that represent specific things, objects, events or concepts. These entity information are used to identify the specific content described by the data. For example, in a government data fusion task for an environmental pollution incident, government agencies need to comprehensively analyze data from different sources in order to have a more comprehensive understanding of the pollution incident and take corresponding measures.

[0134] S30: Determine an optimal sub-data combination based on the R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain a deduction result, and determine a target fusion method based on the deduction result, wherein the deduction result at least includes a stage relationship, the target fusion method at least includes stage fusion, and each element represents one intermediate sub-data;

[0135] In this embodiment, the optimal sub-data combination refers to the combination of the most similar intermediate sub-data. For example, suppose there is a river pollution incident that spans the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties. Therefore, the most similar intermediate sub-data combination must be generated based on these intermediate sub-data to support further analysis and decision-making.

[0136] The optimization algorithm is the particle swarm optimization algorithm. The method of determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm optimization algorithm includes:

[0137] S301: Define the particle's position vector as:

[0138] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw ], each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data;

[0139] The particle swarm optimization algorithm gradually approaches the optimal solution by simulating the movement of particles in a search space. The particle positions represent a vectorized representation of candidate solutions. This embodiment initializes the particle positions to establish the algorithm's search space, enabling the algorithm to begin evaluating and optimizing different sub-data combinations.

[0140] S302: Define the fitness function as:

[0141]

[0142] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, Sim(·) is the similarity function;

[0143] Among them, the fitness function is used as the algorithm optimization target to evaluate the quality of each particle (sub-data combination). The goal is to maximize the similarity of sub-data within the combination. Through the fitness function, the particles are guided to move towards the optimal solution in the search space to ensure that the similarity in the sub-data combination is maximized.

[0144] S303: Update the particle's velocity based on the particle's current velocity, individual optimal position, and global optimal position:

[0145] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0146] Where, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position in the entire particle swarm, p best is the current historical optimal position of particle i;

[0147] Among them, the particle swarm algorithm simulates the movement of particles in the search space by updating the speed, prompting particles to move towards the historical best position and the global best position. Through speed updating, it balances the relationship between exploring new areas and developing existing optimal areas, ensuring that particles can effectively explore the entire search space and avoid falling into local optimal solutions.

[0148] S304: Update the particle's position according to the new velocity:

[0149] x i (t+1)=x i (t)+v i (t+1);

[0150] Where x i (t+1) is the updated position at time t+1, x i (t) is the position of the particle at time t;

[0151] The new position is determined by the current position and velocity, simulating the movement of particles. Updating the position enables particles to move in the search space, thereby approaching or exploring new possible solutions.

[0152] S305: Compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, update the individual optimal position p best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best ;

[0153] This embodiment needs to ensure that each particle continuously learns and updates its position to find the possible optimal solution; the update of the global optimal position ensures that the search direction of the entire particle swarm has a common optimization goal.

[0154] S306: Repeat the above steps S303-S405 until the preset number of iterations is reached and the final global optimal position g best Characterizes the optimal sub-data combination.

[0155] Methods for deducing the relationship between each element in the optimal sub-data combination include:

[0156] The optimal sub-data combination is input into the relational deduction model to obtain the deduction result output by the relational deduction model.

[0157] The methods for constructing the relationship deduction model include:

[0158] A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and historical deduction results, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, and the historical deduction results in the sample training set are used as output data of the regression network, the regression network is trained, and an initial regression network for predicting real-time deduction results is obtained, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model, where the regression network is a deep neural network model.

[0159] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters, weights and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0160] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0161] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0162] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0167] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-source heterogeneous big data processing system, characterized by: include: Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features; Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R; Relationship deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain a deduction result, and determine the target fusion method based on the deduction result. The deduction result includes at least a stage relationship, and the target fusion method includes at least stage fusion, with one element representing one intermediate sub-data; The method for extracting features from unstructured sub-data includes: Encode G elements in the unstructured sub-data to obtain corresponding feature vectors, calculate the semantic similarity between each feature vector, and fuse the feature vectors whose semantic similarity is greater than a preset first similarity threshold to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H; The method for pattern matching based on structured sub-data and multimodal features includes: The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, and the second entity information corresponding to each feature vector in the multimodal feature is obtained. The structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

2. A multi-source heterogeneous big data processing system according to claim 1, characterized in that: Methods for fusing feature vectors include: Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

3. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The method for dividing the data to be processed according to data characteristics includes: Input the data features into the pre-built structural classification model to obtain data classification results, which include structural sub-data and unstructural sub-data; The construction methods of structural classification models include: Obtain Q groups of training data, where Q is a positive integer greater than 1. The training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model.

4. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The optimization algorithm is a particle swarm algorithm. The method for determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm algorithm includes: The first initial unit is used to define the particle's position vector as: ； Among them, each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data; The second initial unit is used to define the fitness function as: ； Where, is the fitness function, It is the index number in the sub-data combination, used to represent each specific element in the sub-data combination. is the index number within the subdata, Indicates that it belongs to The first Sub-data, Indicates that it belongs to The first Sub-data, is the similarity function; Speed ​​update unit: used to update the speed of particles according to the current speed, individual optimal position and global optimal position of particles; Position update unit: used to update the position of the particle according to the new velocity: ； Where, For particles in The updated position at any time, For particles in Position at the moment; Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position is updated. , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position ; Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position Characterizes the optimal sub-data combination.

5. The multi-source heterogeneous big data processing system according to claim 4, characterized in that: The method for updating particle velocity includes: ； Where, For particles exist The speed of time, For particles exist The speed of time, is the inertia weight, is the individual learning factor, is the social learning factor, 、 Both A random number between The current global optimal position in the entire particle swarm, For particles The current best historical position.

6. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The method for performing relationship deduction on each element in the optimal sub-data combination includes: Inputting the optimal sub-data combination into the relational deduction model, thereby obtaining the deduction result output by the relational deduction model; The methods for constructing the relationship deduction model include: A sample data set is obtained, where the sample data set includes a historical optimal sub-data combination and historical deduction results. The sample data set is divided into a sample training set and a sample test set. A regression network is constructed, where the historical optimal sub-data combination in the sample training set is used as input data of the regression network, and the historical deduction results in the sample training set are used as output data of the regression network. The regression network is trained to obtain an initial regression network for predicting real-time deduction results. The initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model.

Images