Multi-source heterogeneous big data processing system

Abstract

The invention relates to the technical field of data processing, and discloses a multi-source heterogeneous big data processing system, which comprises the steps of dividing data into structural sub-data and non-structural sub-data, performing feature extraction on the non-structural sub-data to obtain multi-modal features, so that the data can be compared in a unified feature space, and then, performing feature extraction on the non-structural sub-data to obtain multi-modal features. The similarity and relevance between the data are identified through pattern matching, it is ensured that the data of the same entity are identified to be directly fused, redundancy is reduced, accuracy is improved, then the most similar data combination is selected through an optimization algorithm, and the actual relation type between the data is determined through relation deduction; and finally, the most suitable fusion mode is selected according to the relationships. According to the method, the accuracy and effectiveness of the data fusion process are ensured, and a reliable basis is provided for deeper analysis and decision making.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a multi-source heterogeneous big data processing system. Background Art

[0002] With the rapid development of information technology, the amount of data held by governments at all levels and large enterprises is growing at an exponential rate. However, the diversity of data sources leads to differences in formats, the uneven data quality affects the application effect, the decentralized data management weakens the relevance and integration of data, and the heterogeneity of data forms brings semantic differences and communication barriers. In this context, the task of multi-source heterogeneous data fusion is particularly important. It aims to effectively integrate multimodal data from different sources. At present, most existing research focuses on multimodal deep learning, while there is relatively little research on how to fuse multi-source heterogeneous government data.

[0003] In the prior art, a Chinese patent with publication number CN114817208A provides a method for constructing a multi-source heterogeneous government data organization representation and fusion management model. The patent classifies, cleans, identifies and encrypts government data, and uses metadata parsing and correlation analysis to build a complete data model to achieve efficient storage and management. A Chinese patent with publication number CN116719864A provides a method for fusion of multi-source heterogeneous data in a government cloud. The patent performs noise reduction preprocessing on government data through the standard deviation method, and improves data quality and fusion efficiency based on data similarity classification and heterogeneous data association information mining models.

[0004] Although existing technologies have integrated multi-source heterogeneous government data, there are often different types of relationships between entities in government data, such as same-same relationships and stage relationships. It is difficult for existing technologies to determine a suitable integration method based on these relationships.

[0005] In view of this, the present invention proposes a multi-source heterogeneous big data processing system to solve the above problems. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a multi-source heterogeneous big data processing system.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] In a first aspect, a multi-source heterogeneous big data processing system includes:

[0009] Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features;

[0010] Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R;

[0011] Relationship deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain the deduction result, and determine the target fusion method according to the deduction result. The deduction result at least includes the stage relationship, and the target fusion method at least includes the stage fusion. One element represents one intermediate sub-data.

[0012] Furthermore, the method for extracting features from unstructured sub-data includes:

[0013] The G elements in the unstructured sub-data are encoded to obtain corresponding feature vectors, the semantic similarity between the feature vectors is calculated, and the feature vectors whose semantic similarity is greater than a preset first similarity threshold are fused to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0014] Furthermore, the method of fusing feature vectors includes:

[0015] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0016] Furthermore, the method for pattern matching based on structured sub-data and multimodal features includes:

[0017] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, the second entity information corresponding to each feature vector in the multimodal feature is obtained, and the structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0018] Furthermore, the method of dividing the data to be processed according to data characteristics includes:

[0019] The data features are input into a pre-built structural classification model to obtain data classification results, which include structural sub-data and unstructured sub-data.

[0020] The construction method of the structural classification model includes:

[0021] Get Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model.

[0022] Furthermore, the optimization algorithm is a particle swarm algorithm, and the method for determining the optimal sub-data combination based on the R intermediate sub-data and the particle swarm algorithm includes:

[0023] The first initial unit is used to define the particle's position vector as:

[0024] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw ], each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data.

[0025] The second initial unit is used to define the fitness function as:

[0026]

[0027] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, and Sim(·) is the similarity function.

[0028] Speed ​​update unit: used to update the speed of particles according to the current speed, individual optimal position and global optimal position of particles.

[0029] Position update unit: used to update the position of the particle according to the new velocity:

[0030] x i (t+1)=x i (t)+vi (t+1);

[0031] In the formula, x i (t+1) is the updated position of the particle at time t+1, x i (t) is the position of the particle at time t.

[0032] Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position p is updated. best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best .

[0033] Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

[0034] Furthermore, the method for updating the particle velocity includes:

[0035] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0036] In the formula, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position of the entire particle swarm, p best is the current historical optimal position of particle i.

[0037] Furthermore, the method of deducing the relationship between each element in the optimal sub-data combination includes:

[0038] Inputting the optimal sub-data combination into the relational deduction model, thereby obtaining the deduction result output by the relational deduction model;

[0039] The construction method of the relationship deduction model includes:

[0040] A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and a historical deduction result, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, the historical deduction result in the sample training set is used as output data of the regression network, the regression network is trained to obtain an initial regression network for predicting real-time deduction results, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention divides data into structured sub-data and unstructured sub-data, and performs feature extraction on the unstructured sub-data to obtain multimodal features, so that the data can be compared in a unified feature space. Then, pattern matching is used to identify the similarity and correlation between the data, ensuring that the data of the same entity are directly fused to reduce redundancy and improve accuracy. Next, an optimization algorithm is used to select the most similar data combination, and the actual relationship type between the data is determined through relationship deduction. Finally, the most appropriate fusion method is selected based on these relationships. In this way, the present invention ensures the accuracy and effectiveness of the data fusion process and provides a reliable basis for more in-depth analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of a multi-source heterogeneous big data processing system in the present invention;

[0044] Figure 2 The present invention is a flowchart of a multi-source heterogeneous big data processing method. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example 1

[0047] See also Figure 1 As shown, this embodiment discloses a multi-source heterogeneous big data processing system, including:

[0048] Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features;

[0049] In this embodiment, the data to be processed refers to multi-source heterogeneous data sent by government platforms at all levels or large enterprises. These data may include structured data (such as database records, spreadsheets), unstructured data (such as text files, emails, social media content), and multimodal data (such as images, audio, video). These data come from various sources and in different formats, and usually involve multiple aspects such as government management, public services, and business operations.

[0050] Among them, the data characteristics include at least a structural measurement value and a label density value. The structural measurement value represents the proportion of the data to be processed that conforms to a specific format (such as tabular form, field name, and data type consistency). The label density value represents the ratio of the number of labels in the data to be processed. It can be understood that structured data usually appears in a fixed tabular form, such as rows, columns, field names, and data types (such as SQL databases, spreadsheets), while unstructured data lacks such a fixed format (such as text files, images, audio, and video). Therefore, structured data conforms to a specific format. Structured data usually contains labels and attributes to describe the hierarchical relationship of data elements, while unstructured data usually does not contain such label information. Therefore, the larger the structural measurement value and the label density value, the greater the probability that the data to be processed is structured data.

[0051] Methods for dividing the data to be processed according to data characteristics include:

[0052] The data features are input into a pre-built structural classification model to obtain data classification results, which include structural sub-data and unstructured sub-data.

[0053] The construction method of the structural classification model includes:

[0054] Obtain Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. The historical data features and historical data classification results are used as sample sets, and the sample sets are divided into training sets and test sets. A classifier is constructed, and the historical data features in the training set are used as input data, and the historical data classification results in the training set are used as output data. The classifier is trained to obtain an initial classifier, and the initial classifier is tested using the test set. A classifier that meets a preset accuracy is output as a structural classification model, and the classifier is preferably a naive Bayes model or a support vector machine model.

[0055] Methods for extracting features from unstructured sub-data include:

[0056] The G elements in the unstructured sub-data are encoded to obtain corresponding feature vectors, the semantic similarity between the feature vectors is calculated, and the feature vectors whose semantic similarity is greater than a preset first similarity threshold are fused to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0057] It should be noted that the unstructured sub-data includes G elements, each of which can be text data, image data or video data. When the element is text data, a word embedding model (such as Word2Vec, GloVe) is used to convert the text data into a feature vector. When the element is image data, a convolutional neural network (such as ResNet, Inception) or a visual Transformer (ViT) is used to encode the image and extract a feature vector representing the image content. When the element is video data, the video data can be divided into several image data for processing, or a 3D convolutional neural network (3D-CNN) can be used to encode the video data. The semantic similarity in the above represents the semantic similarity or correlation between different elements (text, image, video). Specifically, the semantic similarity measures whether these different types of data are similar or related in content, theme, concept or semantics.

[0058] It should be added that the method for calculating semantic similarity can be implemented by calculating Euclidean distance or cosine similarity, and the first similarity threshold is pre-set based on expert experience.

[0059] Methods for fusing feature vectors include:

[0060] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0061] It can be understood that when the semantic similarity is greater than the preset second similarity threshold, it indicates that the two feature vectors are in a high similarity situation, the data features are highly similar in semantics and information, and feature splicing can be performed directly to maintain the integrity and relevance of the information. When the semantic similarity is greater than the preset first similarity threshold, it indicates that the two feature vectors are in a medium similarity situation, the data features have a certain degree of complementarity, and the contribution of each feature in the fusion process can be dynamically adjusted through the attention mechanism.

[0062] Among them, feature splicing and fusion is the most direct and commonly used feature-level fusion method, which directly splices feature vectors from different modalities (such as text, image, audio, video, etc.) into a larger feature vector.

[0063] Attention mechanism fusion uses the attention mechanism to dynamically assign weights to feature vectors of different modalities to determine the importance of each feature in the fusion process. This method can flexibly select and focus on the most relevant information and filter out unimportant or noisy features.

[0064] For example, when the feature vector corresponding to the text and the feature vector corresponding to the picture are fused by the attention mechanism, the feature vector corresponding to the text and the feature vector corresponding to the picture need to be input into the attention fusion formula first. The attention fusion formula is specifically:

[0065]

[0066] In the formula, Attention(·) represents the attention mechanism, TFV represents the feature vector corresponding to the text, IFV represents the feature vector corresponding to the image, VK is the value of the feature vector corresponding to the image, Softmax(·) is the probability distribution function, FV, IFV T is the transposed dot product between the feature vector corresponding to the text and the feature vector corresponding to the image, d IFV is the dimension of the feature vector corresponding to the image.

[0067] It should be noted that there is a difference between IFV and VK mentioned above. For example, if there is a picture of an illegal building, then IFV can be the shape and material information of the building's roof, and VK can be the height of the illegal building or the texture information of the illegal building's roof. T It represents the similarity between FV and IFV. The above probability distribution function can convert the calculated similarity into a weight distribution, which is used to weight the feature value of the image. The final Attention (TFV, IFV, VK) is a feature vector that integrates the semantic relationship between text and image. This weighted feature vector can more effectively capture the association information between text and image.

[0068] In this embodiment, due to the diverse types of unstructured data (such as text, images, and videos), directly extracting features from data of different modalities may result in inconsistent features and difficulty in comparison. By encoding each element into a feature vector, different types of data can be mapped to the same feature space so that these data can be compared with each other. Directly extracting features of all unstructured sub-data may result in excessive information, including a large amount of irrelevant or redundant information. By calculating the semantic similarity between feature vectors and only fusing features whose semantic similarity is higher than a threshold, irrelevant features can be filtered out, unnecessary noise can be reduced, and the effectiveness of data fusion can be improved.

[0069] Furthermore, this embodiment does not simply concatenate all similarity features, which may introduce a large amount of irrelevant or redundant information, resulting in increased noise in the fusion result, which in turn reduces the validity of the data. By setting two different similarity thresholds, it is possible to more accurately control which features should be directly concatenated and which require an attention mechanism to determine their relative importance, thereby ensuring the accuracy and relevance of the fusion result and improving the quality of the fused data.

[0070] Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R;

[0071] In this embodiment, pattern matching refers to the use of multimodal features of structured data and unstructured data to identify, compare and associate the same or similar data patterns or entities during the data fusion process. The goal of pattern matching is to find the similarities or associations between data by using structured sub-data (such as database records, spreadsheets, etc.) and multimodal features extracted from unstructured sub-data (such as text, images, videos), and then determine which data belong to the same entity or the same subject or concept. For example, in government data processing, heterogeneous data from multiple sources may be involved, and all data related to an event (such as a text report on environmental pollution, a satellite image of a pollution source, and a video surveillance record) must be matched. Through pattern matching, different modal data describing the same event can be combined together to help more comprehensive understanding and decision-making.

[0072] Methods for pattern matching based on structured sub-data and multimodal features include:

[0073] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, the second entity information corresponding to each feature vector in the multimodal feature is obtained, and the structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0074] It should be noted that entity information refers to the characteristics or identifiers in the data that represent specific things, objects, events or concepts. These entity information are used to identify the specific content described by the data. For example, in a government data fusion task for an environmental pollution incident, government agencies need to comprehensively analyze data from different sources in order to have a more comprehensive understanding of the pollution incident and take appropriate measures.

[0075] In this scenario, structured data (structured sub-data) may come from environmental pollution report records in the database, such as pollution source information, pollutant type, occurrence time, pollution degree, etc.; while unstructured data (unstructured sub-data) may include a news report about the pollution incident, or an image showing the geographical location of the pollution source factory A and its surrounding conditions.

[0076] By using One-Hot encoding or word embedding technology to vectorize the structured sub-data, a feature vector representing the data can be generated. At the same time, the fused multimodal features can be feature representations that integrate text embedding vectors and image feature vectors. Since the first entity information and the second entity information both point to the same entity - "factory pollution", the environmental pollution report records, news reports, and images showing the pollution source factory A can be clustered through clustering methods, thereby combining related data together to facilitate subsequent data fusion and analysis.

[0077] It should be added that the clustering method mentioned above may be through K-means clustering or DBSCAN, etc., which will not be described in detail in this embodiment.

[0078] In this embodiment, by calculating the semantic similarity of the features of the unstructured data and fusing them into multimodal features, and then clustering and matching them with the structured data, this combination strategy can more accurately determine the similarity and correlation between the data. Multimodal feature fusion helps to capture richer information, and the clustering process ensures that only highly related data are clustered together, thereby improving the accuracy and reliability of the data fusion results. In addition, in the unstructured data feature extraction stage, similar feature vectors are merged by calculating semantic similarity, which effectively reduces redundant data. Subsequently, when the structured data and multimodal features are pattern matched and clustered, the processing efficiency is greatly improved because the data volume has been optimized and compressed. Such a combination method can effectively reduce computing costs and improve the overall efficiency of large-scale data processing and fusion.

[0079] It should be noted that, in this embodiment, the data to be fused of the same entity are directly fused to form R intermediate sub-data, and the intermediate sub-data refers to the comprehensive vector after the fusion of the structure sub-vector and the feature vector. In other words, the data to be fused includes the structure sub-vector and the feature vector. Only when the first entity information of the structure sub-vector matches the second entity information of the feature vector, can these data to be fused be fused.

[0080] Since pattern matching will not only cluster the same entities, but also cluster similar entities, and this embodiment needs to perform relationship deduction later, it is necessary to ensure that the data belongs to the same entity. Only under the premise of the same entity can the accuracy and consistency of data fusion be ensured, providing a reliable foundation for subsequent relationship deduction.

[0081] In this context, the data to be fused of the same entity can be directly fused by feature concatenation fusion or weighted average fusion. If weighted average fusion is selected, it is necessary to assign corresponding weights to each vector in advance, for example, the weight of the structure subvector is 0.6 and the weight of the feature vector is 0.4. In this way, the contribution of different types of features to the fusion results can be balanced to more accurately reflect the information of the same entity and provide higher quality data support for subsequent analysis and deduction.

[0082] Relation deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain a deduction result, and determine the target fusion mode according to the deduction result, wherein the deduction result at least includes a stage relationship, the target fusion mode at least includes stage fusion, and one element represents one intermediate sub-data;

[0083] In this embodiment, the optimal sub-data combination refers to the combination of the most similar intermediate sub-data. For example, suppose there is a pollution incident in a river, and the pollution incident crosses the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties. Therefore, the most similar intermediate sub-data combination must be generated based on these intermediate sub-data to support further analysis and decision-making.

[0084] The optimization algorithm is a particle swarm algorithm. The method for determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm algorithm includes:

[0085] The first initial unit is used to define the particle's position vector as:

[0086] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw], each vector represents R groups of sub-data combinations, each group contains W intermediate sub-data;

[0087] The particle swarm optimization algorithm gradually approaches the optimal solution by simulating the movement of particles in the search space. The position of the particle represents the vectorized representation of the candidate solution. This embodiment establishes the search space of the algorithm by initializing the position of the particle, so that the algorithm can start to evaluate and optimize different sub-data combination schemes.

[0088] The second initial unit is used to define the fitness function as:

[0089]

[0090] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, Sim(·) is the similarity function;

[0091] Among them, the fitness function is used as the algorithm optimization target to evaluate the quality of each particle (sub-data combination). The goal is to maximize the similarity of sub-data within the combination. Through the fitness function, the particles are guided to move toward the optimal solution in the search space to ensure that the similarity in the sub-data combination is maximized.

[0092] Speed ​​update unit: used to update the speed of particles according to the current speed, individual optimal position and global optimal position of particles:

[0093] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0094] In the formula, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position of the entire particle swarm, p best is the current historical optimal position of particle i;

[0095] Among them, the particle swarm algorithm simulates the movement of particles in the search space by updating the speed, prompting the particles to move towards the historical best position and the global best position. Through speed updating, it balances the relationship between exploring new areas and developing existing optimal areas, ensuring that particles can effectively explore the entire search space and avoid falling into local optimal solutions.

[0096] Position update unit: used to update the position of the particle according to the new velocity:

[0097] x i (t+1)=x i (t)+v i (t+1);

[0098] In the formula, x i (t+1) is the updated position at time t+1, x i (t) is the position of the particle at time t;

[0099] The new position is determined by the current position and speed, simulating the movement of particles. Updating the position enables particles to move in the search space, thereby approaching or exploring new possible solutions.

[0100] Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position p is updated. best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best ;

[0101] This embodiment needs to ensure that each particle continuously learns and updates its position to find the possible optimal solution; the update of the global optimal position ensures that the search direction of the entire particle swarm has a common optimization goal.

[0102] Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

[0103] This embodiment constructs a fitness function based on the similarity of sub-data combinations and intermediate sub-data, and can accurately evaluate the internal similarity of each combination. In this way, the algorithm can maximize the aggregation of similar sub-data, ensure that the fused data represents the same entity or subject, and avoid the risk of data confusion or mis-fusion. The selection of the optimal sub-data combination can effectively integrate intermediate sub-data from different sources but with the same or similar semantics by maximizing the similarity to form a more comprehensive and coherent data set. This integration provides a richer perspective and a more complete data foundation for subsequent data analysis, relationship deduction and decision-making. Since the fusion of data is based on similarity optimization, it ensures that the data in the combination has similar attributes and characteristics, which provides a more solid foundation for subsequent relationship deduction. Deduction and analysis on such a high-similarity data set can more accurately identify complex relationships between entities and support more in-depth scenario analysis and strategic decision-making.

[0104] Methods for deducing the relationship between each element in the optimal sub-data combination include:

[0105] The optimal sub-data combination is input into the relational deduction model to obtain the deduction result output by the relational deduction model.

[0106] The construction method of the relationship deduction model includes:

[0107] A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and a historical deduction result, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, the historical deduction result in the sample training set is used as output data of the regression network, the regression network is trained to obtain an initial regression network for predicting real-time deduction results, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model, wherein the regression network is a deep neural network model.

[0108] It can be understood that relationship deduction refers to analyzing sub-data combinations to find out the logical relationship, temporal relationship or causal relationship between them. For example, suppose there is a river pollution incident that crosses the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties to generate corresponding sub-data combinations. At this time, the deduction result is a stage relationship. Because when analyzing river pollution incidents, the occurrence, spread and impact of pollution is often a dynamic process, it will go through multiple different development stages. The deduction results can help understand the evolution process and timeline of pollution incidents by revealing the relationship between these stages, thereby better guiding response and decision-making.

[0109] It should be added that, in addition to stage relationships, there are also causal relationships, identical relationships, and hierarchical relationships. For example, in an analysis of a pollution incident, there may be monitoring and response data at different levels such as national, provincial, municipal, and county levels. There is a hierarchical relationship between these data, which reflects the division of responsibilities and information flow between management agencies at all levels. This embodiment does not go into too much detail on this.

[0110] It can be understood that after determining the deduction result in this embodiment, the corresponding target fusion mode is determined according to the corresponding relationship. For example, when the deduction result is a stage relationship, the target fusion mode is stage fusion. Stage fusion refers to the merging and integration of data in stages according to the different stages to which the data belongs during the data fusion process. The data of each stage is fused separately, and then these fusion results are gradually integrated together according to the time or event development sequence. In this way, continuous data reflecting the overall evolution process of the event can be obtained. It should be noted that stage fusion is only a fusion method. The specific fusion technical means can be voting fusion or weighted average fusion, etc. The purpose is to comprehensively and accurately reflect the dynamic change process of the event and provide a logically clear, coherent and orderly data basis for further analysis and decision-making.

[0111] This embodiment divides the data into structured sub-data and unstructured sub-data, and performs feature extraction on the unstructured sub-data to obtain multimodal features, so that the data can be compared in a unified feature space. Then, pattern matching is used to identify the similarity and correlation between the data, ensuring that the data of the same entity is directly fused to reduce redundancy and improve accuracy. Next, an optimization algorithm is used to select the most similar data combination, and relationship deduction is used to determine the actual relationship type between the data. Finally, the most appropriate fusion method is selected based on these relationships. In this way, this embodiment ensures the accuracy and effectiveness of the data fusion process and provides a reliable basis for more in-depth analysis and decision-making.

[0112] Example 2

[0113] See also Figure 2 As shown, based on the same inventive concept, this embodiment discloses a multi-source heterogeneous big data processing method. For details not provided in this embodiment, please refer to the description of the relevant parts in Embodiment 1. The method includes:

[0114] S10: Obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features;

[0115] In this embodiment, the data to be processed refers to multi-source heterogeneous data sent by government platforms at all levels or large enterprises. These data may include structured data (such as database records, spreadsheets), unstructured data (such as text files, emails, social media content), and multimodal data (such as images, audio, video). These data come from various sources and in different formats, and usually involve multiple aspects such as government management, public services, and business operations.

[0116] Methods for dividing the data to be processed according to data characteristics include:

[0117] The data features are input into a pre-built structural classification model to obtain data classification results, which include structural sub-data and unstructured sub-data.

[0118] The construction method of the structural classification model includes:

[0119] Obtain Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. The historical data features and historical data classification results are used as sample sets, and the sample sets are divided into training sets and test sets. A classifier is constructed, and the historical data features in the training set are used as input data, and the historical data classification results in the training set are used as output data. The classifier is trained to obtain an initial classifier, and the initial classifier is tested using the test set. A classifier that meets a preset accuracy is output as a structural classification model, and the classifier is preferably a naive Bayes model or a support vector machine model.

[0120] Methods for extracting features from unstructured sub-data include:

[0121] The G elements in the unstructured sub-data are encoded to obtain corresponding feature vectors, the semantic similarity between the feature vectors is calculated, and the feature vectors whose semantic similarity is greater than a preset first similarity threshold are fused to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

[0122] Methods for fusing feature vectors include:

[0123] Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

[0124] Among them, feature splicing and fusion is the most direct and commonly used feature-level fusion method, which directly splices feature vectors from different modalities (such as text, image, audio, video, etc.) into a larger feature vector.

[0125] Attention mechanism fusion uses the attention mechanism to dynamically assign weights to feature vectors of different modalities to determine the importance of each feature in the fusion process. This method can flexibly select and focus on the most relevant information and filter out unimportant or noisy features.

[0126] For example, when the feature vector corresponding to the text and the feature vector corresponding to the picture are fused by the attention mechanism, the feature vector corresponding to the text and the feature vector corresponding to the picture need to be input into the attention fusion formula first. The attention fusion formula is specifically:

[0127]

[0128] In the formula, Attention(·) represents the attention mechanism, TFV represents the feature vector corresponding to the text, IFV represents the feature vector corresponding to the image, VK is the value of the feature vector corresponding to the image, Softmax(·) is the probability distribution function, FV, IFVT is the transposed dot product between the feature vector corresponding to the text and the feature vector corresponding to the image, d IFV is the dimension of the feature vector corresponding to the image.

[0129] S20: performing pattern matching based on the structured sub-data and the multimodal features, determining N data to be fused, and directly fusion the data to be fused of the same entity to obtain R intermediate sub-data, N ≥ R;

[0130] In this embodiment, pattern matching refers to the use of multimodal features of structured data and unstructured data to identify, compare and associate the same or similar data patterns or entities during the data fusion process. The goal of pattern matching is to find the similarities or associations between data by using structured sub-data (such as database records, spreadsheets, etc.) and multimodal features extracted from unstructured sub-data (such as text, images, videos), and then determine which data belong to the same entity or the same subject or concept. For example, in government data processing, heterogeneous data from multiple sources may be involved, and all data related to an event (such as a text report on environmental pollution, a satellite image of a pollution source, and a video surveillance record) must be matched. Through pattern matching, different modal data describing the same event can be combined together to help more comprehensive understanding and decision-making.

[0131] Methods for pattern matching based on structured sub-data and multimodal features include:

[0132] The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, the second entity information corresponding to each feature vector in the multimodal feature is obtained, and the structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

[0133] It should be noted that entity information refers to the characteristics or identifiers in the data that represent specific things, objects, events or concepts. These entity information are used to identify the specific content described by the data. For example, in a government data fusion task for an environmental pollution incident, government agencies need to comprehensively analyze data from different sources in order to have a more comprehensive understanding of the pollution incident and take appropriate measures.

[0134] S30: determining an optimal sub-data combination based on the R intermediate sub-data and the optimization algorithm, performing relationship deduction on each element in the optimal sub-data combination to obtain a deduction result, and determining a target fusion mode according to the deduction result, wherein the deduction result at least includes a stage relationship, the target fusion mode at least includes stage fusion, and one element represents one intermediate sub-data;

[0135] In this embodiment, the optimal sub-data combination refers to the combination of the most similar intermediate sub-data. For example, suppose there is a pollution incident in a river, and the pollution incident crosses the administrative boundaries of two counties. The government departments of the two counties each have government data related to the pollution incident. After preliminary processing and fusion, these data form their own intermediate sub-data. When conducting government analysis, in order to fully understand the pollution situation and its impact, it is necessary to integrate the intermediate sub-data corresponding to the two counties. Therefore, the most similar intermediate sub-data combination must be generated based on these intermediate sub-data to support further analysis and decision-making.

[0136] The optimization algorithm is a particle swarm algorithm. The method for determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm algorithm includes:

[0137] S301: Define the particle's position vector as:

[0138] [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw ], each vector represents R groups of sub-data combinations, each group contains W intermediate sub-data;

[0139] The particle swarm optimization algorithm gradually approaches the optimal solution by simulating the movement of particles in the search space. The position of the particle represents the vectorized representation of the candidate solution. This embodiment establishes the search space of the algorithm by initializing the position of the particle, so that the algorithm can start to evaluate and optimize different sub-data combination schemes.

[0140] S302: The fitness function is defined as:

[0141]

[0142] In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, Sim(·) is the similarity function;

[0143] Among them, the fitness function is used as the algorithm optimization target to evaluate the quality of each particle (sub-data combination). The goal is to maximize the similarity of sub-data within the combination. Through the fitness function, the particles are guided to move toward the optimal solution in the search space to ensure that the similarity in the sub-data combination is maximized.

[0144] S303: Update the particle's speed according to the particle's current speed, individual optimal position and global optimal position:

[0145] v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)];

[0146] In the formula, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position of the entire particle swarm, p best is the current historical optimal position of particle i;

[0147] Among them, the particle swarm algorithm simulates the movement of particles in the search space by updating the speed, prompting the particles to move towards the historical best position and the global best position. Through speed updating, it balances the relationship between exploring new areas and developing existing optimal areas, ensuring that particles can effectively explore the entire search space and avoid falling into local optimal solutions.

[0148] S304: Update the particle position according to the new velocity:

[0149] x i (t+1)=x i (t)+v i (t+1);

[0150] In the formula, x i (t+1) is the updated position at time t+1, x i (t) is the position of the particle at time t;

[0151] The new position is determined by the current position and speed, simulating the movement of particles. Updating the position enables particles to move in the search space, thereby approaching or exploring new possible solutions.

[0152] S305: Compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, update the individual optimal position p best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best ;

[0153] This embodiment needs to ensure that each particle continuously learns and updates its position to find the possible optimal solution; the update of the global optimal position ensures that the search direction of the entire particle swarm has a common optimization goal.

[0154] S306: Repeat the above steps S303-S405 until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

[0155] Methods for deducing the relationship between each element in the optimal sub-data combination include:

[0156] The optimal sub-data combination is input into the relational deduction model to obtain the deduction result output by the relational deduction model.

[0157] The construction method of the relationship deduction model includes:

[0158] A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and a historical deduction result, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, the historical deduction result in the sample training set is used as output data of the regression network, the regression network is trained to obtain an initial regression network for predicting real-time deduction results, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model, wherein the regression network is a deep neural network model.

[0159] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters, weights and thresholds in the formula are set by technicians in this field according to actual conditions.

[0160] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD) or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0161] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0167] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-source heterogeneous big data processing system, characterized in that: include: Data processing module: used to obtain data features corresponding to the data to be processed, divide the data to be processed into structured sub-data and unstructured sub-data according to the data features, extract features from the unstructured sub-data, and obtain multimodal features; Fusion module: used to perform pattern matching based on structural sub-data and multimodal features, determine N data to be fused, directly fuse the data to be fused of the same entity, and obtain R intermediate sub-data, N ≥ R; Relationship deduction module: used to determine the optimal sub-data combination based on R intermediate sub-data and the optimization algorithm, perform relationship deduction on each element in the optimal sub-data combination to obtain the deduction result, and determine the target fusion method according to the deduction result. The deduction result at least includes the stage relationship, and the target fusion method at least includes the stage fusion. One element represents one intermediate sub-data.

2. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The method for extracting features from unstructured sub-data comprises: The G elements in the unstructured sub-data are encoded to obtain corresponding feature vectors, the semantic similarity between the feature vectors is calculated, and the feature vectors whose semantic similarity is greater than a preset first similarity threshold are fused to generate H feature vectors. The H feature vectors are used as multimodal features, and G>H.

3. The multi-source heterogeneous big data processing system according to claim 2, characterized in that: The method for fusing feature vectors comprises: Determine whether the semantic similarity is greater than a preset second similarity threshold. If so, perform feature concatenation on the feature vectors. If not, perform attention fusion on the feature vectors. The second similarity threshold is greater than the first similarity threshold.

4. The multi-source heterogeneous big data processing system according to claim 2, characterized in that: The method for pattern matching based on structural sub-data and multimodal features includes: The structural sub-data is vectorized to generate a structural vector, the first entity information corresponding to each structural sub-vector in the structural vector is obtained, the second entity information corresponding to each feature vector in the multimodal feature is obtained, and the structural vector and the multimodal feature are clustered according to the first entity information and the second entity information to generate N data to be fused. The structural vector includes P structural sub-vectors, and the multimodal feature includes H feature vectors.

5. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The method for dividing the data to be processed according to data characteristics includes: The data features are input into a pre-built structural classification model to obtain data classification results, which include structural sub-data and unstructured sub-data. The construction method of the structural classification model includes: Get Q groups of training data, where Q is a positive integer greater than 1, and the training data includes historical data features and historical data classification results. Use the historical data features and historical data classification results as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the historical data features in the training set as input data, use the historical data classification results in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a structural classification model.

6. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The optimization algorithm is a particle swarm algorithm. The method for determining the optimal sub-data combination based on R intermediate sub-data and the particle swarm algorithm includes: The first initial unit is used to define the particle's position vector as: [x 11 ,x 12 ,.....,x 1w ],[x 21 ,x 22 ,.....,x 2w ],....,[x R1 ,x R2 ,.....,x Rw ], each vector represents R groups of sub-data combinations, and each group contains W intermediate sub-data. The second initial unit is used to define the fitness function as: In the formula, Fitness(x) is the fitness function, j is the index number in the sub-data combination, which is used to represent each specific element in the sub-data combination, k is the index number in the sub-data, and x jk represents the kth sub-data in the jth sub-data combination, x jl represents the lth sub-data in the jth sub-data combination, and Sim(·) is the similarity function. Speed ​​update unit: used to update the speed of particles according to the current speed, individual optimal position and global optimal position of particles. Position update unit: used to update the position of the particle according to the new velocity: x i (t+1)=x i (t)+v i (t+1); In the formula, x i (t+1) is the updated position of the particle at time t+1, x i (t) is the position of the particle at time t. Comparison unit: used to compare the current fitness value of each particle with its historical best fitness value. If the current fitness value is better, the individual optimal position p is updated. best , compare the individual optimal fitness values ​​of all particles and select the best one as the global optimal position g best . Loop processing unit: repeat the above speed update unit-comparison unit until the preset number of iterations is reached, and the final global optimal position g best Characterizes the optimal sub-data combination.

7. The multi-source heterogeneous big data processing system according to claim 6, characterized in that: The method for updating particle velocity comprises: v i (t+1)=βv i (t)+c1×r1×[p best -x i (t)]+c2×r2×[g best -x i (t)]; In the formula, v i (t+1) is the velocity of particle i at time t+1, v i (t) is the velocity of particle i at time t, β is the inertia weight, c1 is the individual learning factor, c2 is the social learning factor, r1 and r2 are both random numbers between [0,1], g best The current global optimal position of the entire particle swarm, p best is the current historical optimal position of particle i.

8. The multi-source heterogeneous big data processing system according to claim 1, characterized in that: The method for performing relationship deduction on each element in the optimal sub-data combination comprises: Inputting the optimal sub-data combination into the relational deduction model, thereby obtaining the deduction result output by the relational deduction model; The construction method of the relationship deduction model includes: A sample data set is obtained, wherein the sample data set includes a historical optimal sub-data combination and a historical deduction result, the sample data set is divided into a sample training set and a sample test set, a regression network is constructed, the historical optimal sub-data combination in the sample training set is used as input data of the regression network, the historical deduction result in the sample training set is used as output data of the regression network, the regression network is trained to obtain an initial regression network for predicting real-time deduction results, the initial regression network is tested using the sample test set, and the initial regression network that satisfies a preset error value is output as a relationship deduction model.

Images