Multi-modal data dynamic fusion method and system based on distributed edge-cloud cooperation

Abstract

The application provides a multi-modal data dynamic fusion method and system based on distributed edge cloud cooperation, relates to the technical field of data processing, and comprises the following steps: constructing a field knowledge graph, performing generative adversarial completion, performing cross-modal semantic alignment based on an attention mechanism, performing knowledge reasoning, dynamically adjusting a sampling strategy, and cooperatively scheduling sensors. Through semantic enhancement and dynamic sampling strategy optimization, the accuracy and real-time performance of multi-modal data fusion are improved, the system resource consumption is reduced, and efficient data processing under edge cloud cooperation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a multi-modal data dynamic fusion method and system based on distributed edge-cloud collaboration. BACKGROUND

[0002] With the rapid development of Internet of Things, big data and artificial intelligence technology, the collection and processing of multi-source heterogeneous data has become a key component of modern information systems. Multi-modal data fusion technology, as an effective method of integrating data from different sources and different forms, can provide more comprehensive and accurate information, and is widely used in smart cities, industrial internet, intelligent medical care and other fields. Traditional multi-modal data fusion methods mainly rely on centralized architecture, which uniformly transmits various data to the cloud for processing. However, with the exponential increase in the number of edge devices and the size of data, this approach faces problems such as transmission bandwidth bottleneck and insufficient real-time performance. Distributed edge-cloud collaboration architecture provides a new approach to solving these problems by establishing a collaboration mechanism between the edge and the cloud, enabling local processing and efficient fusion of data.

[0003] Existing multi-modal data fusion methods often ignore data missing and incompleteness. In actual applications, due to sensor failure, network transmission anomalies and other reasons, the collected data often has missing values, directly affecting the quality and reliability of the fusion results. Existing methods lack effective data completion mechanisms. Secondly, traditional multi-modal fusion methods usually use simple feature-level or decision-level fusion strategies, failing to fully exploit the semantic associations between different modal data, resulting in fusion results that are difficult to capture complex cross-modal semantic information, reducing the system's understanding ability and application value. In addition, existing multi-source sensor sampling strategies are mostly static and preset, lacking the ability to dynamically adjust according to data value and task requirements, which not only leads to resource waste but also misses critical information, making it difficult to adapt to the precise data collection needs of complex and changing application scenarios.

[0004] In this context, there is an urgent need for a multi-modal data fusion method that can effectively handle data incompleteness, enhance cross-modal semantic understanding, and dynamically adjust data collection strategies according to task requirements, to improve the overall system performance and adaptability. The multi-modal data dynamic fusion method based on distributed edge-cloud collaboration architecture is a technical innovation to solve the above problems. SUMMARY

[0005] The embodiment of the present application provides a multi-modal data dynamic fusion method and system based on distributed edge-cloud collaboration, which can solve the problems in the prior art.

[0006] The first aspect of the embodiment of the present application provides a multi-modal data dynamic fusion method based on distributed edge-cloud collaboration, comprising:

[0007] constructing a domain knowledge graph, performing generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, matching the completed data with semantic concepts in the domain knowledge graph to generate preliminary semantic annotation data;

[0008] performing cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculating semantic correlation between different modal data to obtain a semantic correlation matrix, constructing cross-modal attention weights based on the semantic correlation matrix, and fusing semantic features of different modal data according to the cross-modal attention weights to obtain multi-modal fusion data with enhanced semantics;

[0009] performing knowledge reasoning based on the multi-modal fusion data with enhanced semantics, mining implicit association relationships between data, and updating the implicit association relationships to the domain knowledge graph;

[0010] dynamically adjusting a sampling strategy of multi-source sensors based on a collaborative sampling mechanism of the semantic correlation matrix, establishing a multi-objective optimization framework based on the sampling strategy, solving the multi-objective optimization framework based on a reinforcement learning method to obtain an optimal sampling strategy that meets multi-dimensional constraints, and collaboratively scheduling sampling frequency, sampling accuracy and sampling time of the multi-source sensors according to the optimal sampling strategy;

[0011] collecting data according to the adjusted multi-source sensors, and inputting the collected new data into the generative adversarial completion processing step to realize cyclic iterative optimization of data processing.

[0012] performing generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, and matching the completed data with semantic concepts in the domain knowledge graph to generate preliminary semantic annotation data includes:

[0013] The domain knowledge graph includes a set of semantic concept nodes, a set of concept association edges, and a node adjacency matrix. The association weights between semantic annotation categories are calculated based on the node adjacency matrix, and the association weights are fused with the features of the set of semantic concept nodes to obtain semantic annotation features.

[0014] inputting the semantic annotation features into a generative adversarial completion network, and a generator of the generative adversarial completion network completes missing parts in multi-modal data based on the semantic annotation features;

[0015] The importance of each node in the set of semantic concept nodes in the domain knowledge graph is calculated using the association weights. According to the importance, a semantic annotation category with a discrimination degree higher than a preset discrimination threshold is selected from the set of semantic concept nodes. The selected semantic annotation category and its corresponding semantic annotation features are constructed into a candidate semantic annotation set.

[0016] matching the completion data output by the generative adversarial completion network with the candidate semantic annotation set, obtaining a semantic matching score by calculating cosine similarity between a feature representation of the completion data and semantic annotation features of each semantic annotation category in the candidate semantic annotation set, selecting a most matched semantic annotation category for the completion data based on the semantic matching score, and obtaining preliminary semantic annotation data.

[0017] performing cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculating semantic correlation between different modal data to obtain a semantic correlation matrix, and constructing cross-modal attention weights based on the semantic correlation matrix, including:

[0018] performing feature hierarchical extraction on the preliminary semantic annotation data set by using a multi-scale convolution decomposition operation, performing feature decomposition operation on data of each modality to generate modality feature representation, performing the multi-scale convolution decomposition operation based on the modality feature representation to generate a multi-level semantic feature matrix, and establishing a feature correlation mapping according to the multi-level semantic feature matrix to calculate local attention features;

[0019] performing the multi-scale convolution decomposition operation on the local attention features to establish a feature dependency relationship, extracting semantic correlation information of the local attention features, and generating intra-modal attention features through a multi-scale feature fusion structure; realizing cross-modal semantic mapping by using the multi-scale convolution decomposition operation on the intra-modal attention features of different modalities, and calculating correlation of mapped features in a unified semantic space to obtain a semantic correlation matrix;

[0020] constructing a feature reconstruction loss based on the multi-scale convolution decomposition operation, optimizing the semantic correlation matrix through the feature reconstruction loss to generate an optimized semantic correlation matrix, performing multi-scale feature normalization on the optimized semantic correlation matrix to obtain preliminary attention weights, and fusing the preliminary attention weights with multi-scale modulation features to generate final cross-modal attention weights.

[0021] performing the multi-scale convolution decomposition operation based on the modality feature representation to generate a multi-level semantic feature matrix, and establishing a feature correlation mapping according to the multi-level semantic feature matrix to calculate local attention features, including:

[0022] calculating semantic importance of each position in the modality feature representation, generating a multi-level semantic feature matrix according to the semantic importance, performing multi-scale convolution decomposition operation on the modality feature representation based on the multi-level semantic feature matrix, and generating a multi-level semantic feature set by weighting and combining convolution kernel outputs of different scales through the multi-level semantic feature matrix;

[0023] performing adaptive boundary calibration on the multi-layer semantic feature set, generating hierarchical weight coefficients according to the multi-layer semantic feature matrix, and performing weighted aggregation on the calibrated multi-layer semantic features based on the hierarchical weight coefficients to obtain a multi-layer feature representation;

[0024] constructing a probability-guided feature association using the multi-layer semantic feature matrix, mapping the multi-layer feature representation through feature reconstruction to generate a feature pair, calculating the correlation of the feature pair, and combining the semantic saliency of the corresponding position to obtain an associated feature;

[0025] performing adaptive boundary calibration on the associated feature to obtain an attention weight, and performing weighted combination on the multi-layer feature representation after weight mapping based on the attention weight to obtain a local attention feature.

[0026] Based on the semantic enhanced multi-modal fusion data, knowledge reasoning is performed to mine the implicit association relationship between data, including:

[0027] Constructing a semantic association graph, representing semantic entities as nodes in the semantic association graph, and representing relationships between semantic entities as edges in the semantic association graph;

[0028] Based on the topological structure of the semantic association graph, determine the neighbor node set of each node, calculate the similarity between the initial semantic representation of the node and the initial semantic representation of the neighbor node set to obtain a semantic weight, and perform selective information aggregation on the initial semantic representation of the neighbor node set based on the semantic weight to obtain a semantic aggregated representation of the node;

[0029] Using the semantic aggregated representation to perform multi-hop path search on the semantic association graph, generating a multi-hop reasoning path, each of the multi-hop reasoning paths containing a semantic entity sequence composed of multiple semantic entity nodes, calculating the transition probability between nodes based on the semantic aggregated representation of adjacent nodes in the sequence to obtain a position weight, and performing selective information aggregation on the semantic aggregated representation in the semantic entity sequence according to the position weight to generate a path semantic representation;

[0030] For each pair of semantic entities to be reasoned, a multi-hop reasoning path set is constructed, the matching degree between each path in the multi-hop reasoning path set and the target semantic is calculated according to the path semantic representation to obtain an importance coefficient, and the path semantic representation is subjected to nonlinear transformation and then selective information aggregation to obtain the association relationship between the semantic entity pair.

[0031] Based on the sampling strategy, a multi-objective optimization framework is established, and the multi-objective optimization framework is solved based on a reinforcement learning method to obtain an optimal sampling strategy that satisfies multi-dimensional constraints, including:

[0032] The multi-objective optimization framework comprises a set of optimization objectives and a set of constraint conditions, and generates a set of candidate sampling points based on an initial sampling strategy; a generator of a generative adversarial network receives the set of candidate sampling points, generates an optimized sampling strategy according to the initial sampling strategy, and a discriminator of the generative adversarial network performs feasibility evaluation on the optimized sampling strategy to obtain a strategy quality score;

[0033] The combination term of the set of optimization objectives, the strategy quality score and the constraint term of the set of constraint conditions are adaptively fused to obtain a composite reward function; the composite reward function is used to guide the parameter optimization of the generator, the network parameters of the generator are adjusted according to the strategy quality score output by the discriminator, and an optimized sampling strategy is generated based on the adjusted generator;

[0034] The optimized sampling strategy is used to guide the parameter optimization of the discriminator, the network parameters of the discriminator are adjusted according to the composite reward function, and the optimized sampling strategy is quality evaluated based on the adjusted discriminator;

[0035] The optimized sampling strategy is iteratively optimized based on the composite reward function, the strategy parameters are adjusted by a parameter updating method, and an optimal sampling strategy satisfying multi-dimensional constraints is obtained.

[0036] The initial sampling strategy is used to generate an optimized sampling strategy, and the discriminator of the generative adversarial network is used to perform feasibility evaluation on the optimized sampling strategy to obtain a strategy quality score.

[0037] A set of candidate sampling points containing an initial feasible solution is generated based on an initial sampling strategy, a generator of a generative adversarial network is used to perform quality enhancement on the set of candidate sampling points, an optimized sampling strategy is generated according to the initial sampling strategy, and a discriminator of the generative adversarial network is used to perform feasibility evaluation on the optimized sampling strategy to obtain a strategy quality score.

[0038] In a second aspect of the embodiment of the application, a multi-modal data dynamic fusion system based on distributed edge cloud cooperation is provided, comprising:

[0039] A first unit is configured to construct a domain knowledge graph, perform generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, match the completed data with semantic concepts in the domain knowledge graph, and generate preliminary semantic annotation data.

[0040] The second unit is configured to perform cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculate semantic correlation between different modal data to obtain a semantic correlation matrix, construct cross-modal attention weights based on the semantic correlation matrix, fuse semantic features of different modal data according to the cross-modal attention weights, and obtain multi-modal fusion data with enhanced semantics.

[0041] The third unit is configured to perform knowledge reasoning based on the multi-modal fusion data with enhanced semantics, mine implicit association relationships between data, and update the implicit association relationships to the domain knowledge graph.

[0042] The fourth unit is configured to dynamically adjust a sampling strategy of multi-source sensors based on a collaborative sampling mechanism of the semantic correlation matrix, establish a multi-objective optimization framework based on the sampling strategy, solve the multi-objective optimization framework based on a reinforcement learning method, and obtain an optimal sampling strategy that meets multi-dimensional constraints, and collaboratively schedule sampling frequencies, sampling accuracies and sampling times of the multi-source sensors according to the optimal sampling strategy.

[0043] The fifth unit is configured to collect data according to the adjusted multi-source sensors, and input the collected new data to the generative adversarial completion processing step to realize cyclic iterative optimization of data processing.

[0044] In a third aspect, an electronic device is provided, including:

[0045] a processor;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0048] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0049] The present application has the following advantages:

[0050] By constructing a domain knowledge graph and performing data completion based on a generative adversarial network, and combining cross-modal semantic alignment of the attention mechanism, the problem of large semantic difference and weak correlation between different modal data in the traditional data fusion method is solved, and the accuracy and reliability of multi-modal data fusion are improved.

[0051] A semantic correlation matrix guided collaborative sampling mechanism is adopted to dynamically adjust the sampling strategy of the multi-source sensor, a multi-objective optimization framework is solved based on a reinforcement learning method, efficient allocation of perception resources is realized, data collection costs are reduced, and data quality is ensured.

[0052] A cyclic iteration optimization mechanism for data processing is established, and the implicit association relationship of knowledge reasoning and mining is updated to the domain knowledge graph, forming knowledge accumulation and optimization feedback, thereby effectively improving the adaptability and intelligent decision level of the system to complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A flowchart of a multi-modal data dynamic fusion method based on distributed edge-cloud collaboration of the embodiment of the application is shown in the figure.

[0054] Figure 2 A patent feature extraction flowchart based on multi-level semantic combination of the embodiment of the application is shown in the figure.

[0055] Figure 3 A multi-objective optimization sampling framework based on adversarial enhancement of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0057] The technical scheme of the application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0058] Figure 1 A flowchart of a multi-modal data dynamic fusion method based on distributed edge-cloud collaboration of the embodiment of the application is shown in the figure. Figure 1 As shown in the figure, the method comprises:

[0059] Constructing a domain knowledge graph, performing generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, matching the completed data with semantic concepts in the domain knowledge graph, and generating preliminary semantic annotation data;

[0060] The preliminary semantic annotation data is aligned in a cross-modal semantic manner based on an attention mechanism, a semantic correlation matrix is calculated according to the semantic correlation between different modal data, a cross-modal attention weight is constructed based on the semantic correlation matrix, and the semantic features of different modal data are fused according to the cross-modal attention weight, so as to obtain semantic-enhanced multi-modal fusion data;

[0061] Based on the semantic-enhanced multi-modal fusion data, knowledge reasoning is performed to mine the implicit association relationship between data, and the implicit association relationship is updated to the domain knowledge graph;

[0062] Based on the semantic correlation matrix, a cooperative sampling mechanism is used to dynamically adjust the sampling strategy of the multi-source sensor, a multi-objective optimization framework is established based on the sampling strategy, and the multi-objective optimization framework is solved based on a reinforcement learning method to obtain an optimal sampling strategy that meets multi-dimensional constraints; and the sampling frequency, sampling accuracy and sampling time of the multi-source sensor are cooperatively scheduled according to the optimal sampling strategy;

[0063] According to the adjusted multi-source sensor, data collection is performed, and the collected new data is input into the generative adversarial completion processing step to realize cyclic iterative optimization of data processing.

[0064] In an optional implementation, the generative adversarial completion processing is performed on the multi-modal data based on the domain knowledge graph to obtain completion data, the completion data is matched with the semantic concepts in the domain knowledge graph, and the preliminary semantic annotation data is generated, including:

[0065] The domain knowledge graph includes a set of semantic concept nodes, a set of concept association edges and a node adjacency matrix, the association weight between semantic annotation categories is calculated based on the node adjacency matrix, the association weight is fused with the features of the set of semantic concept nodes to obtain semantic annotation features;

[0066] The semantic annotation features are input into a generative adversarial completion network, and a generator of the generative adversarial completion network completes the missing part in the multi-modal data based on the semantic annotation features;

[0067] The importance of each node in the set of semantic concept nodes in the domain knowledge graph is calculated using the association weight, and a semantic annotation category with a discrimination higher than a preset discrimination threshold is selected from the set of semantic concept nodes according to the importance, and the selected semantic annotation category and its corresponding semantic annotation features are constructed into a candidate semantic annotation set;

[0068] The completion data output by the generative adversarial completion network is matched with the candidate semantic annotation set, a semantic matching score is obtained by calculating the cosine similarity between the feature representation of the completion data and the semantic annotation features of each semantic annotation category in the candidate semantic annotation set, the most matched semantic annotation category is selected for the completion data based on the semantic matching score, and preliminary semantic annotation data is obtained.

[0069] In this embodiment, the construction of the domain knowledge graph is the basis of the entire method, and the domain knowledge graph includes three core components: a semantic concept node set, a concept association edge set, and a node adjacency matrix. Taking the medical image diagnosis field as an example, the semantic concept node set includes disease concept nodes such as "pneumonia", "pulmonary nodule", and "pulmonary emphysema", and each node includes a 300-dimensional semantic feature vector, which is obtained by extracting a pre-trained domain language model. The concept association edge set includes relationship types such as "is a", "co-occurrence", and "similar", and is used to represent the semantic association between different disease concepts. The node adjacency matrix is an N*N matrix (N is the number of nodes), and each element in the matrix represents whether there is an association edge between the corresponding two nodes, 1 if there is, and 0 if there is not. In actual application, the constructed medical knowledge graph includes 500 disease concept nodes and 1200 association edges, forming a knowledge network with rich semantic information.

[0070] The association weights between semantic annotation categories are calculated based on the node adjacency matrix, the row elements of the node adjacency matrix are normalized so that the sum of each row is 1, and a standardized adjacency matrix is obtained. Then, the standardized adjacency matrix is processed using a graph convolution network, and high-order association information between nodes is extracted through two layers of graph convolution operations. The input of the first layer of graph convolution is the standardized adjacency matrix and the initial feature matrix of the node, and the output is the intermediate feature representation; the input of the second layer of graph convolution is the standardized adjacency matrix and the output of the first layer, and finally the node representation containing association information is obtained. ReLU is used as the activation function in the graph convolution network, and the output dimension of each layer is set to 128. Through the above processing, the system obtains an association weight matrix between semantic annotation categories, which reflects the semantic association strength between different disease concepts. For example, the association weight between "pneumonia" and "pulmonary infection" is 0.85, indicating that the semantic association degree between the two concepts is very high.

[0071] The association weight is fused with the features of the set of semantic concept nodes, and an attention mechanism is used to realize this fusion process. For each semantic concept node, the system calculates a weighted sum based on its association weight with other nodes to obtain an association context vector. Then the original feature vector of the node is concatenated with the association context vector, and the fused features are obtained through a fully connected layer mapping. The input dimension of the fully connected layer is the original feature dimension plus the context vector dimension, and the output dimension is 256. The system applies LayerNorm normalization processing to the output of the fully connected layer, and then adds the normalized result to the original feature through a residual connection to finally obtain the semantic annotation feature. This fusion method effectively combines the semantic information of the node itself with the association information in the graph structure, generating a more rich semantic representation. For example, the semantic annotation feature of the "pneumonia" node not only contains its own features, but also fuses the information of the "pulmonary infection", "fever" and other nodes related to it.

[0072] The construction of the generative adversarial completion network is generated, which is composed of a generator and a discriminator. The generator adopts the U-Net architecture, which includes five layers of encoder and five layers of decoder. Each layer of the encoder is composed of a convolution layer, a batch normalization layer and a LeakyReLU activation function, and the convolution kernel size is 4x4 and the step size is 2. Each layer of the decoder is composed of a transposed convolution layer, a batch normalization layer and a ReLU activation function, and is connected to the corresponding encoder layer through a skip connection. The input of the generator includes two parts: multi-modal data with missing parts and semantic annotation features. The system adjusts the dimension of the semantic annotation features through linear projection and injects them into each decoder layer of the generator in the form of a conditional vector to guide the generation process. The discriminator adopts the PatchGAN structure, which includes four convolution layers with a convolution kernel size of 4x4 and a step size of 2, and the last layer outputs the true or false judgment result of each image block. The discriminator also receives the semantic annotation features as conditional input. The network is trained using the Adam optimizer with a learning rate of 0.0002, beta1 of 0.5, beta2 of 0.999, a batch size of 16 and 200 training rounds. The loss function is composed of an adversarial loss, a pixel-level L1 loss and a perception loss, with a weight ratio of 1:10:2. In the medical image completion task, this network can effectively complete the missing areas in the CT image, with a structural similarity (SSIM) of 0.92 and a peak signal-to-noise ratio (PSNR) of 28.3dB.

[0073] The importance of semantic concept nodes in the domain knowledge graph is calculated using the association weight. The PageRank algorithm is used to calculate the importance of each node. For each node, the system calculates its weighted in-degree and weighted out-degree, and the weighted sum of the two is the importance score of the node. In the calculation, the system normalizes the column of the association weight matrix, so that the sum of each column is 1, and then calculates the principal eigenvector by power iteration method, and the iteration number is set to 50 and the convergence threshold is set to 0.0001. The elements of the principal eigenvector obtained are the importance scores of the corresponding nodes.

[0074] The discriminativeness of each semantic annotation category is calculated, and the discriminativeness is defined as the average cosine distance between the category and all other categories (i.e. 1 minus the cosine similarity). According to the importance and discriminativeness calculated, the system selects nodes with high importance (higher than 0.05) and high discriminativeness (higher than 0.3) from the set of semantic concept nodes as candidate semantic annotation categories. In the medical image case, 150 disease categories with high discriminativeness are selected from 500 disease concepts, such as "lung adenocarcinoma", "bronchitis", "pulmonary fibrosis", etc. These categories not only occupy an important position in the knowledge graph, but also have high semantic discriminativeness.

[0075] The completion data output by the generative adversarial completion network is matched with the candidate semantic annotation set, and the completion data is processed using a pre-trained multi-modal feature extractor to obtain its feature representation. The feature extractor uses a ResNet-50 architecture, and the last fully connected layer has an output dimension of 256. Then the cosine similarity between the feature representation and the semantic annotation features of each semantic annotation category in the candidate semantic annotation set is calculated to obtain the semantic matching score. The calculation process of the cosine similarity is to divide the dot product of two vectors by the product of their module lengths, and the value range is between -1 and 1, and the larger the value is, the higher the matching degree is. The system sorts the candidate categories according to the semantic matching score, and selects the category with the highest score as the semantic annotation of the completion data. To improve the reliability of annotation, the system sets the minimum matching threshold to 0.7, and the matching result below the threshold will be marked as the "unknown" category. In a medical image completion task, the system matches the completed lung CT image with the candidate semantic annotation set, and gets the "pneumonia" category corresponding to the highest matching score of 0.86, so the image is annotated as "pneumonia".

[0076] To verify the effectiveness of the method, experiments were conducted on a certain medical image data set containing 10,000 chest CT images, with 20%-40% random missing regions artificially introduced to simulate incomplete data. The method was used to complete and semantically label these images, and compared with three existing methods. The experimental results show that the data completion quality of the method is significantly better than that of the comparison methods, with a PSNR increase of 2.1 dB and a SSIM increase of 0.05; the semantic labeling accuracy reaches 89.3%, which is 7.5 percentage points higher than the second best method. Especially for the case of large missing regions (more than 30%), the advantage of the method is more obvious, and the labeling accuracy remains above 85%, while the comparison methods decrease to below 70%. This result proves that the method has a significant advantage in handling severely missing multi-modal data.

[0077] In practical applications, the method can also be adapted according to the needs of the field, for example, in the field of industrial quality inspection, the knowledge graph can be constructed as a semantic network containing various defect concepts to realize the completion and defect labeling of incomplete industrial images. Experiments show that in a certain electronic component defect detection task, the method can effectively complete the partially occluded component images, and the defect type recognition accuracy reaches 92.1%, providing reliable technical support for automated quality inspection systems.

[0078] In summary, the method provided in the embodiment guides the generation of multi-modal data through the field knowledge graph, and effectively solves the problems of multi-modal data missing and semantic labeling, which has a wide application prospect in the fields of medical image analysis and industrial quality inspection. The method not only improves the data completion quality and semantic labeling accuracy, but also enhances the explainability of the labeling results by introducing the semantic association information of the knowledge graph, providing a solid technical foundation for intelligent analysis systems based on multi-modal data.

[0079] In an optional implementation, the preliminary semantic labeling data is aligned across modalities based on an attention mechanism, a semantic correlation matrix is calculated based on the semantic correlation between different modal data, and a cross-modal attention weight is constructed based on the semantic correlation matrix, including:

[0080] A multi-scale convolution decomposition operation is used to extract features of the preliminary semantic labeling data set in layers, and a modal feature representation is generated for each modality of data by performing a feature decomposition operation; a multi-scale convolution decomposition operation is performed based on the modal feature representation to generate a multi-level semantic feature matrix, and a local attention feature is calculated based on a feature correlation mapping established according to the multi-level semantic feature matrix.

[0081] The multi-scale convolution decomposition operation is performed on the local attention feature to establish a feature dependency relationship, extract semantic correlation information of the local attention feature, and generate an intra-modal attention feature through a multi-scale feature fusion structure; the multi-scale convolution decomposition operation is used on the intra-modal attention features of different modalities to realize cross-modal semantic mapping, and a semantic correlation matrix is obtained by calculating the correlation of the mapped features in a unified semantic space.

[0082] A feature reconstruction loss is constructed based on the multi-scale convolution decomposition operation, and the semantic correlation matrix is optimized through the feature reconstruction loss to generate an optimized semantic correlation matrix; multi-scale feature normalization is performed on the optimized semantic correlation matrix to obtain a preliminary attention weight, and the preliminary attention weight is fused with a multi-scale modulation feature to generate a final cross-modal attention weight.

[0083] In this embodiment, the multi-scale convolution decomposition operation is a core technology for realizing feature hierarchical extraction, and the operation includes three main components: channel separation convolution, deep convolution, and point convolution. The channel separation convolution is responsible for separating the input features along the channel dimension, and each channel is processed independently; the deep convolution extracts features in the spatial dimension while maintaining the number of channels unchanged; and the point convolution performs feature fusion in the channel dimension to realize cross-channel information interaction. In specific implementation, the channel separation convolution uses a 1x1 convolution kernel, the deep convolution uses 3x3, 5x5, and 7x7 convolution kernels of different sizes to extract features of different scales, and the point convolution also uses a 1x1 convolution kernel but the number of output channels can be adjusted. The three convolution operations are connected in series to form a complete multi-scale convolution decomposition unit, and the output of each unit is added to the input through a residual connection to enhance network stability. This technology performs excellently in image feature extraction, with a calculation efficiency about 40% higher and a parameter quantity reduced by more than 50% than standard convolution.

[0084] When performing feature hierarchical extraction on the preliminary semantic annotation dataset, feature decomposition operation is performed on the data of each modality. Taking the visual and text modalities as examples, the visual modality data is processed through three layers of multi-scale convolution decomposition units, the initial input channel number is 3 (RGB image), the first layer outputs 64 channels, the second layer outputs 128 channels, and the third layer outputs 256 channels; the text modality data is first converted into a 300-dimensional vector through an embedding layer, then the context features are extracted through a bidirectional long short-term memory network, and finally processed through two layers of multi-scale convolution decomposition units, with an output of 256 channels. After such processing, the data of the two modalities generates feature representations with modality semantic information. For example, for an image annotated as "red sports car", the visual modality feature captures the shape, color, and texture features of the vehicle, while the text modality feature contains the semantic information of "red" and "sports car".

[0085] When performing a multi-scale convolution decomposition operation based on the modal feature representation to generate a multi-level semantic feature matrix, a three-layer pyramid structure is constructed, each layer using a multi-scale convolution decomposition unit with different parameters, with a convolution kernel size of 3x3, 5x5 and 7x7, a step size of 1, and a padding mode of same, and a LeakyReLU activation function. The bottom layer of the pyramid outputs 64 channels, mainly capturing local detail features; the middle layer of the pyramid outputs 128 channels, capturing medium-scale structural features; and the top layer of the pyramid outputs 256 channels, capturing global semantic features. The features of each layer are spliced after being adjusted to the same spatial resolution through upsampling or downsampling, forming a multi-level semantic feature matrix. Taking an image with a resolution of 224x224 as an example, the resolution of the bottom layer feature is kept at 56x56, the middle layer feature is upsampled to 56x56, and the top layer feature is upsampled to 56x56, finally obtaining a multi-level semantic feature matrix of 56x56x448.

[0086] To calculate the local attention features, a non-local self-attention mechanism is used to model the correlation between features. The specific process is as follows: the multi-level semantic feature matrix is mapped to query, key and value feature spaces through 1x1 convolution, the channel number of query and key features is set to 1 / 8 of the original channel number, and the channel number of value features remains unchanged; the dot product of query and key features is calculated and divided by a scaling factor (the square root of the channel number of key features) to obtain attention scores; the attention scores are normalized by softmax to obtain attention weights; the attention weights are multiplied by the value features and summed to obtain weighted features; finally, the weighted features are mapped back to the original channel number through 1x1 convolution and added to the original features to obtain local attention features. This process enables the system to capture long-range dependencies between different positions in the feature map and enhance the semantic expression ability of the features.

[0087] When performing a multi-scale convolution decomposition operation on the local attention features to establish feature dependency, the system designs a hierarchical feature extraction structure. This structure contains three parallel branches, each branch using a multi-scale convolution decomposition unit with different dilation rates, with dilation rates of 1, 2 and 4, so that the receptive field covers different ranges. The output of each branch is selectively fused through an attention gating mechanism, which generates weights through 1x1 convolution and sigmoid activation function, and weights the features of each branch. This structure can effectively extract multi-scale semantic association information in the local attention features and form a representation with rich contextual dependencies. The processed feature dimension is 56x56x256, retaining the spatial resolution of the original feature but reducing the channel number, which reduces the computational complexity while retaining key semantic information.

[0088] The multi-scale feature fusion structure is designed to generate intra-modal attention features. The structure adopts a variant of the feature pyramid network, which contains two information flow paths: top-down and bottom-up. The top-down path transfers high-level semantic information to low-level features through upsampling and 1x1 convolution. The bottom-up path integrates low-level detailed features into high-level representations through average pooling and 3x3 convolution. The features of the two paths are fused through weighted summation, and the weights are dynamically adjusted by learnable parameters. The system applies a channel attention mechanism to the fused features, extracts channel statistics using global average pooling and global max pooling, generates channel weights through a two-layer fully connected network (the number of neurons in the intermediate layer is 1 / 16 of the number of channels, and the activation function is ReLU), and weights the features to obtain intra-modal attention features. This processing significantly enhances the discriminative ability of the features, and experiments show that the classification accuracy is improved by 4.3 percentage points compared to using the original features directly.

[0089] A multi-scale convolution decomposition operation is used to construct a modal projection network to map attention features of different modalities to a unified semantic space. The network contains two layers of multi-scale convolution decomposition units, with batch normalization and Dropout (rate of 0.2) added in the middle, and the last layer uses a tanh activation function to limit the feature value range to the [-1, 1] interval. For both visual and textual modalities, the system constructs independent projection networks with an output dimension of 128. The mapped features are used to calculate the correlation between elements using cosine similarity, resulting in a semantic correlation matrix with size m x n (m and n are the number of samples for the two modalities, respectively). For example, on a dataset containing 500 images and 1000 text descriptions, the resulting semantic correlation matrix has a size of 500 x 1000, with element values ranging between -1 and 1. The larger the value, the higher the semantic correlation between the corresponding visual and textual samples.

[0090] A feature reconstruction loss is constructed to optimize the semantic correlation matrix. A triplet loss function is designed based on the idea of contrastive learning, which selects multiple negative samples (cross-modal sample pairs with mismatched semantics) for each pair of positive samples (cross-modal sample pairs with matched semantics) to calculate the loss. In specific implementation, the similarity of the positive sample pair should be greater than the similarity of the negative sample pair plus a margin value (set to 0.3). The system also introduces a feature reconstruction constraint, which reconstructs the features of one modality from the features of the other modality, and the reconstruction error is used as an auxiliary loss term. The two losses are combined into a final feature reconstruction loss with a weight ratio of 7:3. The system uses the Adam optimizer to minimize the loss function, with a learning rate of 0.0002, a batch size of 64, and a training round of 100. Through this optimization process, the values of the truly relevant elements in the semantic correlation matrix are enhanced, and the values of the irrelevant elements are suppressed, forming an optimized semantic correlation matrix. On the test set, the average precision of the optimized matrix in the cross-modal retrieval task is improved by 8.7 percentage points.

[0091] The optimized semantic correlation matrix is converted into preliminary attention weights, and a hierarchical normalization strategy is designed, including local normalization and global normalization. The local normalization applies a softmax function to each row or column of the matrix, so that the attention weights of different samples within the same modality to another modality sum to 1. The global normalization applies min-max normalization to the entire matrix, mapping all elements to the interval [0, 1]. The two normalization results are weighted and fused through a learnable parameter α (initial value 0.5) to obtain preliminary attention weights. This hierarchical normalization strategy can maintain the competitiveness within the modalities and the comparability between the modalities, and experiments show that it is 3.2 percentage points higher than a single normalization method in cross-modal matching accuracy.

[0092] The final cross-modal attention weights are generated by fusing multi-scale modulation features, and a multi-scale feature enhancement network is constructed, which contains three parallel branches. Each branch uses different down-sampling rates (1x, 2x, and 4x) to process the original features and capture semantic information at different scales. Down-sampling is achieved through average pooling, and the processed features are restored to the original resolution through deconvolution up-sampling. The features of the three branches are fused through an attention mechanism to generate multi-scale modulation features. The system fuses the preliminary attention weights with the multi-scale modulation features through Hadamard product (element-level multiplication), and then processes them through a 1x1 convolution layer and a sigmoid activation function to generate the final cross-modal attention weights. This fusion process enables the attention weights to have both global semantic correlation and local structural information, significantly improving the accuracy and robustness of cross-modal matching. On test data containing noise and occlusion, the matching accuracy is improved by 6.8 percentage points compared to directly using the semantic correlation matrix.

[0093] The method of the present embodiment has been practically applied in a multimedia content analysis system, and on a multimedia dataset containing 5000 pairs of image-text data, the method achieved a cross-modal retrieval accuracy of 92.4%, which is 7.3 percentage points higher than traditional methods. In particular, for complex semantic samples such as "a boy wearing glasses kicking a football on the grass", which contains multiple semantic entities and relationships, the method can accurately capture the semantic association between the key regions in the image (boy, glasses, football, grass) and the text description, generate high-quality cross-modal attention weights, and guide subsequent semantic understanding and matching tasks. The average time for the system to process a pair of image-text data is 78 milliseconds, meeting the real-time application requirements.

[0094] Through the above technical implementation, the method provided by the present embodiment can effectively solve the challenges in cross-modal semantic alignment, significantly improve the accuracy and efficiency of multi-modal data analysis, and provide strong technical support for intelligent content understanding, cross-modal retrieval, and other applications.

[0095] In an optional implementation, the multi-scale convolution decomposition operation is performed based on the modal feature representation to generate a multi-level semantic feature matrix, and a feature correlation mapping is established according to the multi-level semantic feature matrix to calculate a local attention feature, which includes:

[0096] calculating a semantic importance of each position in the modal feature representation, generating a multi-level semantic feature matrix according to the semantic importance, performing a multi-scale convolution decomposition operation on the modal feature representation based on the multi-level semantic feature matrix, and weighting and combining convolution kernel outputs of different scales through the multi-level semantic feature matrix to generate a multi-layer semantic feature set;

[0097] performing adaptive boundary calibration on the multi-layer semantic feature set, generating a hierarchical weight coefficient according to the multi-level semantic feature matrix, and weighting and aggregating the calibrated multi-layer semantic features based on the hierarchical weight coefficient to obtain a multi-level feature representation;

[0098] constructing a probability-guided feature correlation using the multi-level semantic feature matrix, generating a feature pair through feature reconstruction mapping based on the multi-level feature representation, calculating the correlation of the feature pair, and combining the semantic saliency of the corresponding position to obtain a correlation feature;

[0099] performing adaptive boundary calibration on the correlation feature to obtain an attention weight, weighting and combining the multi-level feature representation after weight mapping based on the attention weight to obtain a local attention feature.

[0100] As shown in Figure 2 , the method includes:

[0101] The semantic importance of each position in the modal feature representation, which is a representation form of the modal data obtained through pre-processing, is calculated. For example, for the image modal, the feature representation dimension is HxWxC, where H and W represent the height and width of the feature map respectively, and C represents the number of channels. The semantic importance calculation process adopts a channel attention mechanism. First, global average pooling and global maximum pooling operations are performed on the feature representation in the spatial dimension, obtaining two 1x1xC feature vectors. These two feature vectors are mapped through two fully connected networks respectively. The first layer reduces the number of channels to C / r (r is the reduction rate, set to 16), and the second layer restores the number of channels to C. The ReLU activation function is used between the fully connected layers, and the last layer outputs the channel attention coefficients with a value range of [0, 1] through the Sigmoid function. The outputs of the average pooling branch and the maximum pooling branch are combined through element-level addition to obtain the final channel attention coefficients. By multiplying the original feature representation with the channel attention coefficients, the preliminary semantic importance of each position is obtained. For example, in the visual-text cross-modal learning task, when processing an image containing "red sports car", the channels related to "red" and "vehicle shape" will obtain higher attention coefficients, such as 0.85 and 0.92, while the background-related channels only have an attention coefficient of 0.3.

[0102] When generating a multi-level semantic feature matrix according to the semantic importance, a three-layer pyramid structure is constructed. For the preliminary semantic importance feature, the system generates a multi-level representation using different scale average pooling operations. The first layer maintains the original resolution to capture fine-grained semantics; the second layer reduces the resolution to 1 / 4 of the original through 2x2 average pooling to capture medium-scale semantics; and the third layer reduces the resolution to 1 / 16 of the original through 4x4 average pooling to capture coarse-grained semantics. The features of each layer are connected after being upsampled to a uniform resolution through bilinear interpolation, forming a multi-level semantic feature matrix of HxWx3C. To enhance information interaction between different levels, the system applies a 1x1 convolution to the connected features for channel fusion, adjusting the number of channels to C to obtain the final multi-level semantic feature matrix. Each position in this matrix contains multiple scale semantic information, which can more comprehensively express the data content. For example, for the "red sports car" image, the fine-grained level captures vehicle details such as headlights and hubcaps, the medium-scale level captures the overall vehicle outline, and the coarse-grained level captures the relationship between the vehicle and the environment.

[0103] When performing multi-scale convolution decomposition operation on modal feature representation based on the multi-level semantic feature matrix, the system designs a multi-branch parallel structure. The structure contains three parallel branches, each branch uses different size of convolution kernel: 3x3, 5x5 and 7x7. To improve computational efficiency, 5x5 and 7x7 convolution kernels are implemented by stacking multiple 3x3 convolutions to achieve equivalent receptive field. The specific implementation of each branch includes: channel separation convolution (1x1 convolution kernel), depth convolution (convolution kernel of corresponding size) and point convolution (1x1 convolution kernel) three components in series. Channel separation convolution is responsible for reducing the number of feature channels to reduce the amount of calculation, depth convolution extracts features in the spatial dimension, and point convolution fuses channel information and restores the original channel number.

[0104] The output features processed by each branch are combined by weighting through the multi-level semantic feature matrix. Specifically, the system uses 1x1 convolution to map the multi-level semantic feature matrix into three groups of weight maps, corresponding to the three convolution branches. Each group of weight maps is normalized by the Softmax function to ensure that the sum of the weights of the three branches is 1. The output features of each branch are multiplied by the corresponding weight map, and then summed to obtain the weighted combined features. This adaptive weighting mechanism enables the system to dynamically adjust the importance of different scale convolution kernels according to the semantic content of the input data, improving the relevance of feature extraction. After three layers of such multi-scale convolution decomposition structure, the system generates three layers of semantic features, forming a multi-level semantic feature set.

[0105] Performing adaptive boundary calibration on the multi-level semantic feature set, due to the large difference in feature value distribution caused by multi-scale convolution, the system designs an adaptive boundary calibration mechanism to unify the feature distribution. In specific implementation, the system first calculates the global statistical information of each layer of features, including mean, standard deviation, maximum and minimum. Then based on these statistics, the system constructs an adaptive normalization function to map the feature values to a predetermined range [-1, 1]. The mapping process takes into account the skewness of the feature distribution, using nonlinear mapping for distributions with large skewness and linear mapping for features close to normal distribution. The calibration process also includes a learnable scaling and bias parameter to further optimize the feature distribution. The calibrated multi-level semantic features retain the original semantic information while having a more uniform numerical range, which is beneficial for subsequent feature fusion.

[0106] The hierarchical weight coefficients are generated from the multi-level semantic feature matrix, and the calibrated features are weighted and aggregated to obtain the multi-level feature representation. The importance of each layer feature is calculated using the attention mechanism. Specifically, the multi-level semantic feature matrix is compressed into a vector through global average pooling, and then the hierarchical weight coefficients are generated through two fully connected networks. The first layer of the fully connected network reduces the input dimension to half and uses ReLU activation; the second layer outputs a dimension equal to the number of semantic feature layers, and uses the Softmax function to ensure that the sum of the weights is 1. The obtained hierarchical weight coefficients are used to weighted sum the calibrated multi-level semantic features, generating a multi-level feature representation that integrates information from each layer. For example, when processing clear close-range object images, the fine-grained level obtains a weight of 0.6, while the medium-scale and coarse-grained levels obtain weights of 0.3 and 0.1, respectively; when processing blurred long-range images, the coarse-grained level obtains a higher weight.

[0107] A feature reconstruction mapping module is designed, which includes two parallel projection branches, each consisting of two fully connected networks. The input is the multi-level feature representation, and the output dimensions are the same. The features generated by the two branches constitute a feature pair, which is used for subsequent correlation calculation. The system calculates the cosine similarity between the feature pairs to obtain the original correlation matrix. At the same time, the system extracts the semantic saliency of each position from the multi-level semantic feature matrix. The method is to apply global average pooling and 1x1 convolution to the feature matrix, and then pass it through a Sigmoid function to obtain a saliency map with a value range of [0, 1]. Multiply the original correlation matrix by the semantic saliency at the corresponding position to obtain the weighted correlation considering the importance of semantics. This weighting mechanism ensures that the correlation of the semantic important region is given more attention. Finally, the system applies the Softmax function to normalize the weighted correlation to obtain the final association feature. For example, when processing a scene containing multiple objects, the main object region will have a higher semantic saliency (such as 0.9), so its feature association will have a larger proportion in the final result.

[0108] An adaptive boundary calibration is performed on the association feature to generate high-quality attention weights. An adaptive boundary calibration mechanism similar to the previous one is used, but it is optimized for the characteristics of the association feature. Specifically, the system analyzes the distribution characteristics of the association feature, calculates its statistics, and then designs a suitable mapping function to convert the association feature into attention weights. Unlike before, the system introduces a temperature parameter to adjust the smoothness of the mapping in this step. The initial value of the temperature parameter is set to 1.0, which is automatically optimized through backpropagation. The smaller the temperature value, the steeper the distribution after mapping, highlighting the high correlation region; the larger the temperature value, the smoother the distribution after mapping, reducing the difference between different regions. The calibrated association feature is passed through a Sigmoid function to obtain the final attention weight, with a value range of [0, 1].

[0109] The multi-level feature representation is weighted and combined based on the attention weight after weight mapping to obtain a local attention feature. A double-path attention mechanism is designed: one path adjusts the channel dimension of the multi-level feature representation through 1x1 convolution to generate a feature value; the other path also generates an attention modulation factor through 1x1 convolution, and the final attention map is obtained by multiplying the attention modulation factor and the previously calculated attention weight. Then, the feature value and the attention map are multiplied to realize weighted combination, and a local attention feature is obtained. To further enhance the feature expression capability, the system also introduces a residual connection to add the original multi-level feature representation and the weighted combination result to obtain the final output. The residual connection helps to alleviate the gradient disappearance problem and ensure the training stability of the deep network.

[0110] In the visual-text cross-modal learning task, the method is applied to process a data set containing 2000 pairs of image-text data, the image resolution is 224x224, and the average length of the text is 25 words. For the image modality, the initial feature dimension is 28x28x512; for the text modality, the initial feature dimension is 25x512, representing a 512-dimensional representation of 25 words. After processing by the method, the dimension of the local attention feature remains unchanged, but the semantic expression capability is significantly enhanced. In the cross-modal retrieval task, compared with the traditional method, the retrieval accuracy of image to text is improved by 8.7 percentage points, and the retrieval accuracy of text to image is improved by 7.5 percentage points. In particular, for samples containing complex scenes and multiple semantics, the advantages of the method are more obvious, such as for describing a complex scene "a girl wearing glasses is holding a brown dog on the grass", the method can accurately capture the key semantic areas (girl, glasses, dog, grass) in the image, and realize accurate cross-modal matching.

[0111] Through the above technical implementation, the method provided by the embodiment can effectively process modal feature representation and generate high-quality local attention features, laying a solid foundation for subsequent cross-modal semantic alignment and matching tasks. The method exhibits excellent performance in multiple cross-modal understanding tasks, proving the effectiveness and advancement of its technical solution.

[0112] In an optional implementation, knowledge reasoning is performed based on the semantic-enhanced multi-modal fusion data, and the implicit association relationship between data is mined, including:

[0113] A semantic association graph is constructed, and semantic entities are represented as nodes in the semantic association graph, and the relationship between semantic entities is represented as an edge in the semantic association graph;

[0114] The topology of the semantic correlation graph is used to determine a set of neighbor nodes for each node, a similarity between an initial semantic representation of the node and initial semantic representations of the set of neighbor nodes is calculated to obtain a semantic weight, and the initial semantic representations of the set of neighbor nodes are selectively aggregated based on the semantic weight to obtain a semantic aggregated representation of the node;

[0115] Multi-hop path search is performed on the semantic correlation graph using the semantic aggregated representation to generate multi-hop reasoning paths, each of which contains a semantic entity sequence composed of a plurality of semantic entity nodes, a transition probability between nodes is calculated based on semantic aggregated representations of adjacent nodes in the sequence to obtain a position weight, and the semantic aggregated representations in the semantic entity sequence are selectively aggregated according to the position weight to generate a path semantic representation;

[0116] A set of multi-hop reasoning paths is constructed for each pair of semantic entities to be reasoned, a matching degree between each path in the set of multi-hop reasoning paths and a target semantic is calculated based on the path semantic representation to obtain an importance coefficient, and the path semantic representation is nonlinearly transformed and selectively aggregated to obtain an association relationship between the pair of semantic entities.

[0117] In the step of constructing a semantic correlation graph, semantic entities are represented as nodes in the graph, and relationships between semantic entities are represented as edges in the graph. Semantic entities include specific concepts, things, or abstract concepts, such as "heart disease", "high blood pressure", "aspirin", etc. in the medical field. Semantic relationships represent interactions, causal relationships, or subordinate relationships between entities, such as "treatment", "cause", "containment", etc. The system assigns an initial semantic representation to each semantic entity node, which is a 768-dimensional vector extracted by a pre-trained language model, used to capture the semantic information of the entity. Each edge in the graph contains two main attributes: relationship type and relationship strength. The relationship type is selected from a predefined set of relationship types, which contains 45 relationship types; the relationship strength is a real number between 0 and 1, representing the confidence of the relationship. Taking a medical knowledge graph as an example, there is a "treatment" relationship between "aspirin" and "heart disease", with a relationship strength of 0.85, indicating that the confidence of this treatment relationship is high. The complete semantic correlation graph constructed by the system contains about 25,000 semantic entity nodes and 120,000 relationship edges, forming a rich semantic knowledge network.

[0118] The semantic similarity between node v and each of its neighbor nodes u is calculated by the system to obtain a semantic weight. The semantic similarity is calculated using the cosine similarity method, i.e., the cosine value of the angle between the initial semantic representation vectors of the two nodes is calculated. The calculation process is to divide the dot product of the two vectors by the product of their module lengths. The semantic weight obtained ranges from -1 to 1, and the greater the value, the higher the semantic similarity. The system performs softmax normalization processing on the obtained semantic weight, so that the sum of all weights is 1. For example, the normalized semantic weights of the "hypertension" node and its neighbor nodes "heart disease", "kidney disease", and "sodium salt" are 0.5, 0.3, and 0.2, respectively, indicating that the semantic correlation between "hypertension" and "heart disease" is the strongest.

[0119] Based on the semantic weight, the initial semantic representation of the neighbor node set is selectively aggregated to generate a node semantic aggregation representation. The weighted summation method is used to realize information aggregation, i.e., the initial semantic representation vector of each neighbor node is multiplied by the corresponding semantic weight, and then the summation is obtained to obtain an aggregation vector. Further, the system fuses the aggregation vector with the initial semantic representation vector of the center node. The specific way is to concatenate the two vectors and then map them back to the original dimension through a fully connected layer, and then pass through a ReLU activation function to obtain the final semantic aggregation representation. The fully connected layer parameters are set as follows: the input dimension is twice the original dimension (concatenation result), the output dimension is the same as the original dimension, and the weight is initialized using the Xavier initialization method. In this way, the semantic representation of each node not only contains its own information, but also fuses the relevant semantic information of the neighbor nodes, and the obtained semantic aggregation representation dimension remains 768, but the semantics is more rich. For example, the semantic aggregation representation of the "hypertension" node integrates its own semantic information and the associated information of its neighbor nodes such as "heart disease" and "kidney disease", forming a more comprehensive semantic expression.

[0120] The multi-hop reasoning path is generated by multi-hop path search using semantic aggregated representation. A bidirectional search strategy is adopted, which searches from both the source node and the target node, and the maximum hop limit is 4. The beam search algorithm is used in the search process, and the beam width is set to 50, that is, the top 50 paths with the highest scores are retained at each step. The path score is calculated based on the similarity between the semantic aggregated representation of each node in the path and the target query. The specific method is to calculate the cosine similarity between the average vector of the semantic aggregated representation of all nodes in the path and the target query vector. The system filters the generated paths and retains the paths with high semantic coherence and query relevance. Each retained multi-hop reasoning path contains a series of semantic entity nodes, forming a semantic entity sequence. For example, for the pair of entities "aspirin" and "stroke", a multi-hop reasoning path generated by the system is "aspirin-anticoagulation effect-thrombosis-stroke", which indicates that aspirin prevents stroke by its anticoagulation effect to prevent thrombosis.

[0121] The transition probability between nodes is calculated based on the semantic aggregated representation of adjacent nodes in the sequence to determine the position weight. For each pair of adjacent nodes u and v in the path, the system calculates the transition probability, which is the cosine similarity between the semantic aggregated representation of node u and the semantic aggregated representation of node v, and then mapped to 0 to 1 through the sigmoid function. The transition probability represents the rationality of transferring from node u to node v, and the larger the value, the more reasonable the transition. The calculation of the position weight takes into account the position information of the node in the path, and is realized by using the attention mechanism. The system constructs the query vector (the concatenation of the semantic representations of the start and end points of the path) and the semantic aggregated representation of each position in the path to calculate the attention score. The specific method is to take the dot product of the query vector and the semantic aggregated representation of each position, and then normalize it through the softmax function to obtain the position weight. For example, in the path "aspirin-anticoagulation effect-thrombosis-stroke", the position weights of the two intermediate nodes "anticoagulation effect" and "thrombosis" are 0.3 and 0.4 respectively, indicating that "thrombosis" contributes more to the understanding of the relationship between aspirin and stroke.

[0122] The semantic aggregation representation in the semantic entity sequence is selectively information converged to generate a path semantic representation according to a position weight. The semantic aggregation representations of the nodes in the path are weighted and summed according to the position weight to obtain an initial representation of the path. Then, the initial representation is spliced with a representation of the path relationship sequence (obtained by average pooling on all relationship representations in the path), and a final path semantic representation is obtained by mapping through a multi-layer perception. The structure of the multi-layer perception is: the input layer dimension is the initial representation dimension plus the relationship sequence representation dimension, the hidden layer dimension is 3 / 4 of the input dimension, the output layer dimension is 512, the hidden layer activation function is ReLU, and the output layer does not use the activation function. The final path semantic representation is a 512-dimensional vector that captures the overall semantic information expressed by the path. For example, the semantic representation of the path "aspirin-anticoagulation-blood clots-stroke" combines the semantic information of the nodes in the path and their position importance, expressing the overall semantic "aspirin prevents stroke by preventing blood clots through anticoagulation".

[0123] A set of multi-hop reasoning paths is constructed for each pair of reasoning semantic entities to achieve comprehensive reasoning. The system sets an upper limit of 20 paths for each pair of entities, and preferentially selects paths with high scores. For the pair of entities "aspirin" and "stroke" in the medical knowledge base, the system constructs a set of multi-hop reasoning paths, including: path 1: "aspirin-anticoagulation-blood clots-stroke"; path 2: "aspirin-anti-inflammatory-blood vessel inflammation-stroke"; path 3: "aspirin-platelet aggregation inhibition-blood clots-stroke", etc. Each path explains the association between aspirin and stroke from a different perspective, providing multi-dimensional evidence for subsequent reasoning of the association.

[0124] The importance coefficient of each path in the set of multi-hop reasoning paths is determined according to the matching degree between the path semantic representation and the target semantic, and the target semantic is represented by the query intent, which can be a predefined relationship type vector such as "treatment", "prevention", etc. The system calculates the cosine similarity between the semantic representation of each path and the target semantic vector to obtain the original matching score. Then, the matching scores of all paths are normalized by the softmax function to obtain the importance coefficients of each path, so that the sum of the importance coefficients of all paths is 1. For example, for the query "how does aspirin prevent stroke", the importance coefficients of paths 1, 2, and 3 are 0.6, 0.1, and 0.3, respectively, indicating that path 1 provides the most consistent explanation of the prevention relationship.

[0125] The association relationship between the semantic entity pairs is obtained by selectively aggregating the information of the path semantic representations after nonlinear transformation. The nonlinear transformation is implemented by a two-layer fully connected network. The input dimension of the first layer is 512, the output dimension is 256, and the activation function is ReLU. The input dimension of the second layer is 256, the output dimension is 128, and no activation function is used. In the information aggregation process, the transformed representations of each path are weighted and summed according to the importance coefficients to obtain the final vector representation representing the association relationship between the entity pairs. The system calculates the similarity between this vector and the pre-defined relationship type representation, and selects the relationship type with the highest similarity as the prediction result. At the same time, the system can extract the most contributing path as the explanation basis. For example, the system infers that there is a "prevention" relationship between "aspirin" and "stroke" with a confidence of 0.85, and the main basis is the reasoning path "aspirin prevents thrombosis through anticoagulation to prevent stroke".

[0126] In the medical knowledge reasoning application instance, the system constructs a semantic association graph containing 25,000 medical entities and 120,000 relationship edges. Through the above technical means, the system can answer complex queries such as "whether a certain drug is effective for a certain disease" and "whether there is a concurrent relationship between two diseases", with an accuracy of 87.5%, which is 15.3 percentage points higher than traditional methods. The system can provide an interpretable reasoning path such as "ibuprofen reduces inflammation in arthritis by inhibiting prostaglandin synthesis", enhancing the credibility and explainability of the reasoning results. The average time for the system to process a complex medical query is 0.8 seconds, meeting the real-time application requirements.

[0127] In the financial risk association analysis scenario, the method is successfully applied to construct a semantic association graph containing 12,000 financial entities and 95,000 relationships. The system can accurately identify hidden fund flow paths between enterprises, such as "enterprise A--legal person B--enterprise C--offshore account D", helping regulatory agencies discover potential financial risks. In the financial association entity identification task, the accuracy of the system is 83.6%, the recall rate is 79.2%, and the F1 score is 81.3%, which is 18.5 percentage points higher than traditional rule-based methods. The system can complete multi-path reasoning of complex financial entity relationships within 5 seconds, meeting the efficiency requirements of regulatory agencies.

[0128] Through the above embodiments, the multi-modal fusion data knowledge reasoning method based on semantic enhancement provided by the technical scheme can effectively mine the implicit association relationships between data, provide high-quality reasoning results and interpretable reasoning paths, and demonstrate superior application value in medical, financial and other fields.

[0129] In an optional embodiment, a multi-objective optimization framework is established based on the sampling strategy, the multi-objective optimization framework is solved based on a reinforcement learning method, and an optimal sampling strategy satisfying multi-dimensional constraints is obtained, including:

[0130] The multi-objective optimization framework includes an optimization objective set and a constraint condition set, a candidate sampling point set is generated based on an initial sampling strategy, a generator of a generative adversarial network receives the candidate sampling point set, an optimized sampling strategy is generated according to the initial sampling strategy, a discriminator of the generative adversarial network is used to perform feasibility evaluation on the optimized sampling strategy to obtain a strategy quality score;

[0131] The combined terms of the optimization objective set, the strategy quality score, and the constraint terms of the constraint condition set are adaptively fused to obtain a composite reward function; the composite reward function is used to guide the parameter optimization of the generator, the network parameters of the generator are adjusted according to the strategy quality score output by the discriminator, and an optimized sampling strategy is generated based on the adjusted generator;

[0132] The optimized sampling strategy is used to guide the parameter optimization of the discriminator, the network parameters of the discriminator are adjusted according to the composite reward function, and the optimized sampling strategy is quality evaluated based on the adjusted discriminator;

[0133] The optimized sampling strategy is iteratively optimized based on the composite reward function, the strategy parameters are adjusted by a parameter updating method, and an optimal sampling strategy satisfying multi-dimensional constraints is obtained.

[0134] As shown in Figure 3 , the method includes:

[0135] An optimization objective set and a constraint condition set are determined, the optimization objective set includes three indexes of sampling coverage, sampling efficiency, and data representativeness. The sampling coverage calculation method is the proportion of key area sampling points, with a value range of 0-1; the sampling efficiency calculation method is the number of sampling points completed per unit time; the data representativeness is calculated by the ratio of sample variance to population variance, and the ideal value is 1. The constraint condition set includes resource limitation, time limitation, and spatial distribution constraint, wherein the resource limitation is defined as the total sampling points not exceeding a preset threshold N; the time limitation is that the total sampling time does not exceed T hours; the spatial distribution constraint requires that the distance between adjacent sampling points is not less than D meters. When the system initializes the sampling strategy, a grid sampling method is used to uniformly distribute MxM candidate sampling points in the target area, and each sampling point records its position coordinates (x, y) and sampling priority p value, with the initial value of p being 0.5.

[0136] When constructing the generative adversarial network architecture, the generator network adopts a four-layer fully connected structure, the input layer dimension is the number of candidate sampling points x 3 (corresponding to the x coordinate, y coordinate and priority p), 128 neurons are set in the first hidden layer, 64 neurons are set in the second hidden layer, and the output layer dimension is the same as the input layer. ReLU activation function is used between each layer, and Sigmoid function is used in the last layer to map the output to the [0, 1] interval. When the generator propagates forward, the candidate sampling point set information is converted into a feature vector input into the network, and after four layers of calculation, the optimized sampling point coordinate adjustment and priority adjustment are output, and the original value is added to obtain the optimized sampling strategy parameters.

[0137] The discriminator network adopts a three-layer fully connected structure design, the input layer dimension is the same as the output dimension of the generator, the first hidden layer sets 96 neurons, the second hidden layer sets 32 neurons, and the output layer is a single neuron. The hidden layer uses Leaky ReLU activation function, and the output layer uses Sigmoid function to output the quality score between 0 and 1. When the discriminator evaluates, the optimized sampling strategy parameter vector is input into the network, and after three layers of calculation, the strategy quality score q is output, and the higher the score, the better the strategy quality. When the discriminator is initialized, pre-training is performed using historical high-quality strategy samples, the number of samples is 500, the number of training rounds is 200, the Adam optimizer is used, and the learning rate is set to 0.001.

[0138] In the construction process of the composite reward function, the optimization target item, the strategy quality score item and the constraint penalty item are combined. The optimization target item adopts linear weighting, the coverage weight is 0.4, the efficiency weight is 0.3, and the representative weight is 0.3. The strategy quality score item directly uses the discriminator output value q, and the weight is set to 0.5. The constraint penalty item calculates the violation degree and converts it into a penalty value for each constraint condition, and the initial value of the weight is set to 0.5, which is dynamically adjusted with the constraint violation degree. Specifically, the resource constraint penalty value is max(0, n-N) / N, n is the actual number of sampling points; the time constraint penalty value is max(0, t-T) / T, t is the actual sampling time; the space constraint penalty value is the number of point pairs that violate the minimum distance requirement divided by the total number of point pairs. The three parts are summed by weighted sum to get the final composite reward value R, and after the calculation is completed, R is normalized to the interval [-1, 1].

[0139] The generator parameter optimization is implemented using a policy gradient method. In each iteration, the generator generates 32 batches of sampling policy samples, and the corresponding composite reward value R is calculated for each sample. The system calculates the gradient direction to adjust the network parameters in the direction of increasing R value. The Adam optimizer is used for parameter update, with an initial learning rate of 0.001 and a weight decay coefficient of 0.0001. The gradient clipping threshold is set to 5.0 to prevent gradient explosion. The current policy performance is evaluated every 100 rounds during optimization. If the performance improvement is less than 0.01 for three consecutive evaluations, the learning rate is halved. To enhance exploration ability, Gaussian noise is added to the output layer when generating the policy, with an initial noise standard deviation of 0.2 that decays linearly to 0.01 with the increase of iteration number.

[0140] During the discriminator parameter optimization process, the system constructs a training data set containing positive and negative samples. The positive samples include historical high-quality policies and artificially designed expert strategies, with a label of 1. The negative samples include low-quality policies generated by the generator, with a label linearly mapped to the [0, 0.7] interval according to the composite reward value R. The training data set size is 200, with a positive to negative sample ratio of 1:4. The discriminator uses a binary cross-entropy loss function and is optimized using the stochastic gradient descent method, with a learning rate of 0.0005 and a momentum coefficient of 0.9. The batch size is 32. Randomly select 80% of the samples in each batch for training to prevent overfitting. The discriminator is verified every 5 batches, and if the verification set accuracy is lower than the previous one for three consecutive times, the training is stopped in advance.

[0141] The optimal sampling strategy iteration optimization uses an alternating training mechanism. The generator generates a batch of sampling strategies, with a batch size of 32. The discriminator evaluates the quality score of each strategy. The composite reward value of each strategy is calculated. The generator parameters are updated. The discriminator parameters are updated every 5 rounds. The system maintains an experience replay pool with a capacity of 1000, storing historical generated strategies and their reward values. 80% of the samples come from the current batch and 20% of the samples come from the replay pool randomly selected. The iteration process sets an early stopping mechanism. When the composite reward value improvement is not more than 0.001 for 50 consecutive iterations, it is considered that the optimization converges, and the current optimal sampling strategy is output. The optimal strategy is saved in JSON format, containing sampling point coordinates, priority and sampling order information.

[0142] In practical application scenarios, sampling optimization is performed for quality detection of an industrial production line. The initial candidate sampling points are set as 20x10 grid, a total of 200 evenly distributed detection positions, and each position records coordinates and an initial priority of 0.5. The constraint conditions are set as: the number of sampling points is not more than 60, the detection time is not more than 30 minutes, and the distance between adjacent sampling points is not less than 0.5 meters. The optimization objectives are set as: the defect detection rate is not less than 99.5%, the detection time is the shortest, and the sample representativeness is the best. The system training parameters are set as: the total number of iterations is 5000, the generator learning rate is 0.001, the discriminator learning rate is 0.0005, and the initial value of each weight in the compound reward function is 0.5.

[0143] After training, the final optimized sampling strategy contains 58 sampling points, the detection time is 27.3 minutes, and the defect detection rate reaches 99.7%. The characteristics of the optimized sampling points are: high sampling density in areas with high historical defect rates, sampling point spacing greater than 0.5 meters, and minimum total sampling path length. Compared with the initial uniform sampling strategy, the detection time is reduced by 64% while maintaining the same defect detection rate. The system also records the sampling point priority ranking to guide the order in the actual detection process, further improving the detection efficiency.

[0144] When the adaptive adjustment mechanism is implemented, the system updates the defect probability distribution map every 7 days. The update method is: record the number of defects detected in each area, update the probability map using the exponential weighted moving average method, the weight of recent data is 0.7, and the weight of historical data is 0.3. When a new defect pattern is detected, the system increases the probability value of the region by 50%; when there is no defect detection in a certain area for 30 consecutive days, the corresponding sampling density is reduced by 30%. After each update, the system re-runs the optimization algorithm based on the new probability distribution map, with 1000 iterations, and reuses the previously trained generator and discriminator parameters as initial values to accelerate convergence.

[0145] When the transfer learning technology is implemented, the first two layers of the generator and the first two layers of the discriminator are saved for rapid adaptation to new environments. In the new scenario, the system loads the saved parameters to initialize the low-level network, and only randomly initializes the subsequent layer parameters with a learning rate of 1 / 10 of the original value, i.e., 0.0001 for the generator and 0.00005 for the discriminator. During training, the first two layers of parameters remain unchanged, and only the subsequent layer parameters are updated, with 1400 iterations. Test results show that after using transfer learning, the convergence speed of the new scenario strategy optimization is 3.5 times that of complete retraining, and the final strategy performance is equivalent.

[0146] When dealing with high-dimensional sampling space, the system uses an autoencoder for dimension reduction. The structure of the autoencoder is as follows: the dimension of the input layer is equal to the original feature dimension, the encoder contains two hidden layers with dimensions of 1 / 2 and 1 / 4 of the original dimension respectively, and the latent space dimension is set to 8; the decoder is symmetrically designed and restored to the original dimension through two hidden layers. The autoencoder is trained for 1000 rounds using the mean square error loss function with a learning rate of 0.001. The optimization process is performed in an 8-dimensional latent space, and after optimization, it is mapped back to the original space through the decoder. In the 32-dimensional sampling feature space test, the system calculates the reconstruction error as 3.2%, and the final strategy performance is only 2.1% lower than that of direct optimization, but the computing efficiency is improved by 8.6 times, and the optimization time is shortened from 75 hours to 8.7 hours.

[0147] In an optional implementation, the method further comprises: generating an optimized sampling strategy according to the initial sampling strategy; and performing feasibility evaluation on the optimized sampling strategy by using a discriminator of the generative adversarial network to obtain a strategy quality score.

[0148] The method further comprises: generating a candidate sampling point set containing an initial feasible solution based on the initial sampling strategy; performing quality enhancement on the candidate sampling point set by using a generator of the generative adversarial network; and generating an optimized sampling strategy according to the initial sampling strategy.

[0149] The method further comprises: generating a candidate sampling point set containing an initial feasible solution based on the initial sampling strategy; in actual implementation, the Monte Carlo method can be used to randomly sample candidate points from the initial strategy distribution. For example, assuming that the initial sampling strategy is defined as a uniform distribution in a spatial region, the system can randomly generate 1000 sampling points in the region as a candidate set. These sampling points need to satisfy basic constraint conditions, such as being located within the effective boundary in the physical space or satisfying the predetermined value range limit in the parameter space.

[0150] To ensure the quality of the initial feasible solution, the system also performs preliminary screening on each candidate sampling point. For example, for a robot path planning problem, the initial sampling point needs to avoid known obstacle regions; for a network resource allocation problem, the initial sampling point needs to satisfy basic resource capacity constraints. After screening, the system retains 800 candidate sampling points that satisfy the basic constraints.

[0151] The generator of the generative adversarial network is used to enhance the quality of the candidate sample point set. The generator receives the candidate sample points as input and extracts and converts features through a multi-layer neural network. In practical applications, the generator network structure can include multiple fully connected layers, each followed by batch normalization and ReLU activation functions. For example, a typical generator network can include an input layer (with the same dimension as the sample points), three hidden layers (each with 256 neurons), and an output layer (with the same dimension as the sample points).

[0152] During the quality enhancement process, the generator learns to map the initial sample points to a better quality solution space. For example, for a point [0.25, 0.42, 0.18] in the initial candidate sample point set, the generator converts it to [0.27, 0.45, 0.16], making it closer to the optimal solution area. Through iterative optimization, the generator gradually improves the quality of the sample points and generates a new sample point set closer to the true optimal solution distribution.

[0153] During the quality enhancement process, the system also introduces a diversity preservation mechanism to avoid the generated sample points being overly concentrated in certain local areas. This can be achieved by adding a diversity regularization term to the loss function of the generator. For example, by calculating the distance between generated samples, the system encourages the samples to be distributed. After processing, the system obtains a quality-enhanced sample point set, with a size of the original candidate set, such as 800 points.

[0154] Based on the quality-enhanced sample point set, the embodiment generates an optimized sampling strategy according to the initial sampling strategy. This is achieved through density estimation techniques, such as kernel density estimation or Gaussian mixture models. For example, the system can input the enhanced 800 sample points into a kernel density estimator, use a Gaussian kernel function with a bandwidth parameter of 0.05, and generate a continuous probability density function as the optimized sampling strategy.

[0155] The new sampling strategy generated in this way will tend to produce sample points in high-quality areas while retaining appropriate exploration capabilities. For example, in the original strategy, the space is uniformly sampled, while the optimized strategy assigns higher sampling probabilities to areas that are expected to yield better results.

[0156] The discriminator of the generative adversarial network is used to evaluate the feasibility of the optimized sampling strategy and obtain a strategy quality score. The discriminator network is usually composed of multiple fully connected layers, such as an input layer, three hidden layers (each with 128 neurons using Leaky ReLU activation), and an output layer (a single neuron using Sigmoid activation).

[0157] A batch of test points (e.g., 200 points) are randomly sampled from the optimized sampling strategy; these test points are input into the discriminator to obtain the feasibility score of each point. For example, for the sampling point [0.27, 0.45, 0.16], the discriminator gives a feasibility score of 0.87, indicating that this point has an 87% probability of being a feasible high-quality solution.

[0158] The strategy quality score is calculated as the average of the feasibility scores of all test points. For example, if the average score of 200 test points is 0.75, the optimized sampling strategy is considered to have a quality level of 75%. The system can set a threshold (e.g., 0.7) to determine whether the strategy is acceptable. If the strategy quality score is lower than the threshold, the system will return to the quality enhancement step and perform a new round of optimization.

[0159] In a practical application case, the initial sampling strategy of a resource scheduling problem is to uniformly sample in the resource allocation space. After optimization by this method, the system generates a strategy that tends to allocate more resources to high-demand nodes, and the strategy quality score is improved from the initial 0.53 to 0.82, significantly improving resource utilization and system performance.

[0160] Through the above embodiments, this method realizes the automatic optimization and quality evaluation of the sampling strategy, effectively improves the sampling efficiency and the quality of the solution, and is suitable for solving various complex optimization problems.

[0161] In a second aspect, the embodiment of the present application provides a multi-modal data dynamic fusion system based on distributed edge-cloud cooperation, comprising:

[0162] A first unit is configured to construct a domain knowledge graph, perform generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, match the completed data with semantic concepts in the domain knowledge graph, and generate preliminary semantic annotation data;

[0163] A second unit is configured to perform cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculate semantic correlation between different modal data to obtain a semantic correlation matrix, construct a cross-modal attention weight based on the semantic correlation matrix, fuse semantic features of different modal data according to the cross-modal attention weight, and obtain semantic-enhanced multi-modal fusion data;

[0164] A third unit is configured to perform knowledge reasoning based on the semantic-enhanced multi-modal fusion data, mine implicit association relationships between data, and update the implicit association relationships to the domain knowledge graph;

[0165] The fourth unit is configured to dynamically adjust a sampling strategy of the multi-source sensor based on a cooperative sampling mechanism of the semantic correlation matrix, establish a multi-objective optimization framework based on the sampling strategy, solve the multi-objective optimization framework based on a reinforcement learning method, and obtain an optimal sampling strategy meeting multi-dimensional constraints; and the sampling frequency, sampling accuracy and sampling time of the multi-source sensor are cooperatively scheduled according to the optimal sampling strategy.

[0166] The fifth unit is configured to collect data from the adjusted multi-source sensor, input the collected new data into the generative adversarial completion processing step, and realize cyclic iterative optimization of data processing.

[0167] In a third aspect, an electronic device is provided, and the electronic device includes:

[0168] a processor;

[0169] a memory for storing processor-executable instructions;

[0170] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0171] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0172] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0173] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal data dynamic fusion method based on distributed edge-cloud collaboration, characterized in that, The method comprises the following steps: constructing a domain knowledge graph, performing generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, matching the completed data with semantic concepts in the domain knowledge graph to generate preliminary semantic annotation data; performing cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculating semantic correlation between different modal data to obtain a semantic correlation matrix, constructing cross-modal attention weights based on the semantic correlation matrix, and fusing semantic features of different modal data according to the cross-modal attention weights to obtain semantic-enhanced multi-modal fusion data; performing knowledge reasoning based on the semantic-enhanced multi-modal fusion data, mining implicit association relationships between data, and updating the implicit association relationships to the domain knowledge graph, including: constructing a semantic association graph, representing semantic entities as nodes in the semantic association graph, and representing relationships between semantic entities as edges in the semantic association graph; determining a neighbor node set of each node based on the topological structure of the semantic association graph, calculating the similarity between the initial semantic representation of the node and the initial semantic representation of the neighbor node set to obtain a semantic weight, and selectively aggregating the initial semantic representation of the neighbor node set based on the semantic weight to obtain a semantic aggregated representation of the node; performing multi-hop path search on the semantic association graph using the semantic aggregated representation to generate multi-hop reasoning paths, each of the multi-hop reasoning paths comprising a semantic entity sequence composed of multiple semantic entity nodes, calculating the transition probability between nodes based on the semantic aggregated representation of adjacent nodes in the sequence to obtain a position weight, and selectively aggregating the semantic aggregated representation in the semantic entity sequence according to the position weight to generate a path semantic representation; constructing a multi-hop reasoning path set for each pair of semantic entities to be reasoned, calculating the matching degree between each path in the multi-hop reasoning path set and the target semantics to obtain an importance coefficient, and selectively aggregating the path semantic representation after nonlinear transformation to obtain the association relationship between the semantic entity pair; dynamically adjusting the sampling strategy of the multi-source sensor based on the collaborative sampling mechanism of the semantic correlation matrix, establishing a multi-objective optimization framework based on the sampling strategy, solving the multi-objective optimization framework based on a reinforcement learning method, and obtaining an optimal sampling strategy that satisfies multi-dimensional constraints; and cooperatively scheduling the sampling frequency, sampling accuracy and sampling time of the multi-source sensor according to the optimal sampling strategy; collecting data according to the adjusted multi-source sensor, and inputting the collected new data into the generative adversarial completion processing step to realize cyclic iterative optimization of data processing.

2. The method of claim 1, wherein, performing generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, matching the completed data with semantic concepts in the domain knowledge graph to generate preliminary semantic annotation data, including: The domain knowledge graph includes a semantic concept node set, a concept association edge set, and a node adjacency matrix. An association weight between semantic annotation categories is calculated based on the node adjacency matrix. The association weight is fused with features of the semantic concept node set to obtain semantic annotation features. The semantic annotation features are input into a generative adversarial completion network. A generator of the generative adversarial completion network completes missing parts in multi-modal data based on the semantic annotation features. The importance of each node in the semantic concept node set in the domain knowledge graph is calculated using the association weight. According to the importance, a semantic annotation category with a discrimination higher than a preset discrimination threshold is selected from the semantic concept node set. The selected semantic annotation category and its corresponding semantic annotation features are constructed into a candidate semantic annotation set. The completion data output by the generative adversarial completion network is matched with the candidate semantic annotation set. A semantic matching score is obtained by calculating the cosine similarity between the feature representation of the completion data and the semantic annotation features of each semantic annotation category in the candidate semantic annotation set. Based on the semantic matching score, the most matched semantic annotation category is selected for the completion data to obtain preliminary semantic annotation data.

3. The method of claim 1, wherein, Based on the attention mechanism, cross-modal semantic alignment is performed on the preliminary semantic annotation data. A semantic correlation matrix is calculated based on the semantic correlation between different modal data. Cross-modal attention weights are constructed based on the semantic correlation matrix. Multi-scale convolution decomposition operations are used to extract features of the preliminary semantic annotation data set in layers. Modal feature representations are generated by performing feature decomposition operations on data of each modality. Based on the modal feature representations, multi-level semantic feature matrices are generated by performing the multi-scale convolution decomposition operations. Local attention features are calculated based on the feature association mapping of the multi-level semantic feature matrices. The multi-scale convolution decomposition operations are performed on the local attention features to establish feature dependency relationships. Semantic association information of the local attention features is extracted. Intra-modal attention features are generated by a multi-scale feature fusion structure. Cross-modal semantic mapping is achieved by using the multi-scale convolution decomposition operations on the intra-modal attention features of different modalities. The semantic correlation matrix is calculated based on the correlation between the mapped features in a unified semantic space. Based on the multi-scale convolution decomposition operations, a feature reconstruction loss is constructed. The semantic correlation matrix is optimized to generate an optimized semantic correlation matrix by the feature reconstruction loss. Preliminary attention weights are obtained by performing multi-scale feature normalization on the optimized semantic correlation matrix. The preliminary attention weights are fused with multi-scale modulation features to generate final cross-modal attention weights.

4. The method of claim 3, wherein, Based on the modal feature representations, multi-level semantic feature matrices are generated by performing the multi-scale convolution decomposition operations. Local attention features are calculated based on the feature association mapping of the multi-level semantic feature matrices. The semantic importance of each position in the modal feature representation is calculated, and a semantic probability distribution matrix is generated according to the semantic importance; a multi-scale convolution decomposition operation is performed on the modal feature representation based on the semantic probability distribution matrix, and the convolution kernel outputs of different scales are weighted and combined through the semantic probability distribution matrix to generate a multi-layer semantic feature set; An adaptive boundary calibration is performed on the multi-layer semantic feature set, a hierarchical weight coefficient is generated according to the semantic probability distribution matrix, and a multi-layer feature representation is obtained by weighted aggregation of the calibrated multi-layer semantic features based on the hierarchical weight coefficient; A probability-guided feature association is constructed using the semantic probability distribution matrix, the multi-layer feature representation is mapped to generate a feature pair through feature reconstruction, the correlation of the feature pair is calculated, and the associated features are obtained by combining the semantic saliency of the corresponding position; An adaptive boundary calibration is performed on the associated features to obtain an attention weight, and the multi-layer feature representation is weighted and combined based on the attention weight after weight mapping to obtain a local attention feature.

5. The method of claim 1, wherein, A multi-objective optimization framework is established based on the sampling strategy, and a reinforcement learning method is used to solve the multi-objective optimization framework to obtain an optimal sampling strategy that satisfies multi-dimensional constraints, including: The multi-objective optimization framework includes an optimization objective set and a constraint condition set, and a candidate sampling point set is generated based on an initial sampling strategy; a generator of a generative adversarial network receives the candidate sampling point set, generates an optimized sampling strategy based on the initial sampling strategy, and a discriminator of the generative adversarial network is used to evaluate the feasibility of the optimized sampling strategy to obtain a strategy quality score; The combined terms of the optimization objective set, the strategy quality score, and the constraint terms of the constraint condition set are adaptively fused to obtain a composite reward function; the composite reward function is used to guide the parameter optimization of the generator, the network parameters of the generator are adjusted according to the strategy quality score output by the discriminator, and an optimized sampling strategy is generated based on the adjusted generator; The optimized sampling strategy is used to guide the parameter optimization of the discriminator, the network parameters of the discriminator are adjusted according to the composite reward function, and the quality of the optimized sampling strategy is evaluated based on the adjusted discriminator; The optimized sampling strategy is iteratively optimized based on the composite reward function, and the strategy parameters are adjusted through a parameter updating method to obtain an optimal sampling strategy that satisfies multi-dimensional constraints.

6. The method of claim 5, wherein, Generating an optimized sampling strategy based on the initial sampling strategy, and using a discriminator of a generative adversarial network to evaluate the feasibility of the optimized sampling strategy to obtain a strategy quality score, including: A candidate sampling point set containing an initial feasible solution is generated based on an initial sampling strategy, a generator of a generative adversarial network is used to enhance the quality of the candidate sampling point set, and an optimized sampling strategy is generated based on the initial sampling strategy; a discriminator of the generative adversarial network is used to evaluate the feasibility of the optimized sampling strategy to obtain a strategy quality score.

7. A multi-modal data dynamic fusion system based on distributed edge-cloud collaboration, for implementing the method of any one of the preceding claims 1-6, characterized in that, Including: The first unit is configured to construct a domain knowledge graph, perform generative adversarial completion processing on multi-modal data based on the domain knowledge graph to obtain completed data, match the completed data with semantic concepts in the domain knowledge graph, and generate preliminary semantic annotation data. The second unit is configured to perform cross-modal semantic alignment on the preliminary semantic annotation data based on an attention mechanism, calculate semantic correlation between different modal data to obtain a semantic correlation matrix, construct cross-modal attention weights based on the semantic correlation matrix, fuse semantic features of different modal data according to the cross-modal attention weights, and obtain semantic-enhanced multi-modal fusion data. The third unit is configured to perform knowledge reasoning based on the semantic-enhanced multi-modal fusion data, mine implicit association relationships between data, and update the implicit association relationships to the domain knowledge graph. The fourth unit is configured to dynamically adjust a sampling strategy of a multi-source sensor based on a collaborative sampling mechanism of the semantic correlation matrix, establish a multi-objective optimization framework based on the sampling strategy, solve the multi-objective optimization framework based on a reinforcement learning method, and obtain an optimal sampling strategy that satisfies multi-dimensional constraints; and cooperatively schedule a sampling frequency, a sampling accuracy, and a sampling time of the multi-source sensor according to the optimal sampling strategy. The fifth unit is configured to collect data according to the adjusted multi-source sensor, and input the collected new data to the generative adversarial completion processing step to realize cyclic iterative optimization of data processing.

8. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method of any one of claims 1-6. The computer program instructions are executed by the processor to implement the method of any one of claims 1-6. ​ ​ 9. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​

Images