Domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration

Abstract

The invention relates to a domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration, and relates to the technical field of machine learning domain generalization. The method comprises the following steps: independently training on each source domain to obtain a source domain model, and extracting and learning semantic features of local data; carrying out weighted average on parameters of each source domain model on different network levels, introducing semantic similarity measurement to carry out model weight distribution, and aggregating each semantic feature; and carrying out cross-level alignment calibration operation on the aggregated semantic features by utilizing an attention mechanism, carrying out performance evaluation on the fused source domain model, and adjusting the semantic calibration intensity according to a performance evaluation result. Key semantic features are extracted and fused from a plurality of source domains, so that the generalization performance of the model can be enhanced; due to the fact that the source domain model is independently trained on each source domain and semantic aggregation is carried out in a data-independent mode, direct sharing of original data is avoided, and data privacy can be protected.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning domain generalization, and in particular to a domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration. Background Art

[0002] In the field of machine learning, domain generalization technology is dedicated to improving the generalization ability of the model for unseen target domains. In practical applications, model training often involves data from multiple source domains, which may not be directly shared due to the need for privacy protection. Therefore, how to effectively use multi-source data to improve the generalization ability of the model without leaking privacy has become an urgent problem to be solved. Existing domain generalization technologies mostly focus on improving the generalization ability of a single source domain, or simply merge multiple source domain data for processing, often ignoring the importance of data privacy, failing to fully utilize the hierarchical semantic information of multi-source data, and failing to effectively solve the problem of data distribution differences between source domains. In the process of multi-source data fusion, existing technologies lack protection for data privacy, which can easily cause user privacy leakage during model training. For example, some methods based on adversarial training, such as conditional adversarial domain adaptation (CADA), have made progress in dealing with the distribution differences between the source domain and the target domain, but usually require access to unlabeled data in the target domain, which is often not feasible in practical applications.

[0003] Therefore, traditional domain generalization technology assumes that all source domain data can be centrally accessed and processed, which poses a great risk in terms of privacy protection; that is, when processing multi-source data, data privacy is often not taken into consideration, and there are dangerous problems caused by data leakage. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration are provided, which can protect data privacy and avoid the risks caused by data leakage.

[0005] A domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration, the method comprising: Acquire source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of local data; The parameters of each source domain model are weighted averaged at different network levels, and a semantic similarity metric is introduced to distribute model weights to obtain a fused source domain model, and each semantic feature is aggregated through the fused source domain model; The attention mechanism is used to perform cross-level alignment and calibration operations on the aggregated semantic features to obtain calibrated and optimized features. Based on the calibration optimization feature, the performance of the fused source domain model is evaluated, and the strength of the semantic calibration is adjusted according to the performance evaluation result to obtain a target source domain model.

[0006] In one embodiment, the source domain data is obtained, and a source domain model is obtained by independent training on each source domain, including: Acquire various source domain data, and use the various source domain data to perform model training; Feature representations in each of the source domain data are extracted respectively, and the classification loss on the source domain is minimized based on each of the feature representations to complete the training of the source domain model.

[0007] In one embodiment, weighted averaging of the parameters of each source domain model at different network levels includes: Determine the total number of source domains and the weight of each source domain based on the source domain data; Respectively obtaining parameters of each of the source domain models at the target network level; A weighted average calculation is performed according to the total number of source domains, source domain weights, and parameters to obtain fused model parameters of the target network level.

[0008] In one embodiment, a semantic similarity metric is introduced to perform model weight assignment to obtain a fused source domain model, including: Determining a reference model, and calculating a semantic similarity measure between the reference model and the source domain data at the target network level; Obtaining a similarity control coefficient, and calculating a weight allocation strategy according to the similarity control coefficient and a semantic similarity metric; The model weights are distributed according to the weight distribution strategy, the connection weights between different network layers are adjusted, and the differential privacy technology is introduced to obtain the fused source domain model.

[0009] In one embodiment, an attention mechanism is used to perform cross-level alignment and calibration operations on aggregated semantic features, including: Determine a feature dimension of the aggregated semantic feature, obtain an attention weight of the source domain data on the feature dimension, and determine a mapping function corresponding to the feature dimension; Calculating an attention-weighted feature representation according to the fused model parameters, attention weights, and mapping functions; Perform cross-layer feature alignment on the attention-weighted feature representation to obtain aligned features.

[0010] In one embodiment, the method further comprises: Determine the total number of network levels, and calculate the feature average of all semantic features of the target network level; The domain invariance loss is calculated based on the total number of network layers and the feature average.

[0011] In one embodiment, based on the calibration optimization feature, a performance evaluation is performed on the fused source domain model, and the strength of the semantic calibration is adjusted according to the performance evaluation result to obtain a target source domain model, including: Determine a validation set on the target domain, input the validation set on the target domain into the fused source domain model, and obtain the current performance of the fused source domain model; Comparing the current performance with the reference performance to obtain a comparison result; The dynamic calibration strength is calculated according to the comparison result to obtain the target source domain model.

[0012] In one embodiment, the method further comprises: Applying the dynamic calibration strength to a source domain model and obtaining the domain invariance loss; Calculating a weighted calibration loss according to the dynamic calibration strength and the domain invariance loss; The weighted calibration loss is applied to the source domain model and the target source domain model is obtained by optimization.

[0013] A domain generalization system based on multi-source hierarchical semantic fusion and cross-layer calibration, the system comprising: A local semantic acquisition module is used to acquire source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of local data; A semantic aggregation module, used to perform weighted averaging of the parameters of each source domain model at different network levels, and introduce semantic similarity measurement to distribute model weights, so as to obtain a fused source domain model, and aggregate various semantic features through the fused source domain model; The cross-layer semantic calibration module is used to use the attention mechanism to perform cross-layer alignment and calibration operations on the aggregated semantic features to obtain calibrated optimized features; The dynamic semantic calibration strategy module is used to perform performance evaluation on the fused source domain model based on the calibration optimization feature, and adjust the strength of the semantic calibration according to the performance evaluation result to obtain a target source domain model.

[0014] The above-mentioned domain generalization method and system based on multi-source hierarchical semantic fusion and cross-layer calibration can extract and fuse key semantic features from multiple source domains by independently training the source domain model on each source domain and fusing the source domain models, thereby enhancing the generalization performance of the model; since the source domain model is trained independently on each source domain and semantic aggregation is performed in a data-independent manner, direct sharing of the original data is avoided, data privacy can be protected, and risks caused by data leakage can be avoided; and the intensity of the semantic calibration can be dynamically adjusted according to the model performance evaluation results, which can make the model training process more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an application environment diagram of a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration in one embodiment; Figure 2 is a flowchart of a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration in one embodiment; Figure 3 is a structural block diagram of a domain generalization system based on multi-source hierarchical semantic fusion and cross-layer calibration in one embodiment; Figure 4 A schematic diagram of the structure of a local semantics acquisition module in one embodiment; Figure 5 This is a schematic diagram of the structure of a semantic aggregation module in one embodiment; Figure 6 A schematic diagram of the structure of a cross-layer semantic calibration module in one embodiment; Figure 7 A schematic diagram of the structure of a dynamic semantic calibration strategy module in an embodiment; Figure 8 A schematic diagram of an end-to-end model optimization process in one embodiment; Fig. 9 Schematic diagram of a signaling structure of a domain generalization method in an embodiment; Fig.10 It is a flowchart of an intelligent recommendation system applied to multi-source data fusion in one embodiment; Fig.11 A schematic diagram of a process of cross-modal fusion diagnosis applied to medical images in one embodiment; Fig.12 A schematic diagram of a process for detecting surface defects of industrial products in one embodiment; Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can obtain source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of local data; the computer device 110 can perform weighted average of the parameters of each source domain model at different network levels, and introduce semantic similarity measurement to distribute model weights to obtain a fused source domain model, and aggregate various semantic features through the fused source domain model; the computer device 110 can use the attention mechanism to perform cross-level alignment and calibration operations on the aggregated semantic features to obtain calibration optimization features; the computer device 110 can perform performance evaluation on the fused source domain model based on the calibration optimization features, and adjust the strength of semantic calibration according to the performance evaluation results to obtain the target source domain model. Among them, the computer device 110 can be, but is not limited to, various personal computers, laptops, smart phones, robots, unmanned aerial vehicles and other devices.

[0018] In one embodiment, Figure 2 As shown, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, including the following steps: Step 202, source domain data is obtained, source domain models are independently trained on each source domain, and corresponding source domain data are optimized through the source domain model to extract and learn semantic features of local data.

[0019] Computer equipment can obtain source domain data, independently train models on each source domain, extract and learn the semantic information of local data, and optimize specific loss functions so that the model can capture the distribution characteristics of the data, laying the foundation for subsequent semantic aggregation.

[0020] Specifically, in one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of independently training a model, and the specific process includes: obtaining data from each source domain, and using each source domain data for model training; respectively extracting feature representations from each source domain data, minimizing the classification loss on the source domain based on each feature representation, and completing the training of the source domain model.

[0021] In each source domain D hOn the other hand, the computer equipment can independently train a source domain model M h , the source domain model is optimized specifically for its corresponding source domain data, and the training goal of the model is to minimize the classification loss on the source domain while maintaining sensitivity to the data semantics. The objective function of model training can be expressed as: ;in, represents the cross entropy loss function, It is model M h The predicted probability that an input sample x is from class y.

[0022] After completing the source domain model training, the computer device can further extract semantic information. Specifically, the model M h Through a series of convolutional layers and pooling layers, high-level feature representations of the input data are extracted. These feature representations not only capture the local features of the data, but also contain rich semantic information. The feature extraction process can be described as: Among them, F h Represents the source domain D h The extracted features, φ represents the model M h feature extraction operation.

[0023] In this embodiment, in order to further optimize the source domain model's ability to capture semantic information, an additional contrast loss function is introduced to enhance the source domain model's understanding of the intrinsic structure of the data. Specifically, the formula for contrast loss can be expressed as: ;in, is a function that measures the distance between feature vectors. , denotes the eigenvectors, represents the ReLU activation function, ensuring that the loss is non-negative; λ con is the regularization coefficient, It is model M h A collection of parameters.

[0024] In step 204, the parameters of each source domain model are weighted averaged at different network levels, and a semantic similarity metric is introduced to distribute the model weights to obtain a fused source domain model, and various semantic features are aggregated through the fused source domain model.

[0025] Computer equipment can fuse the parameters of different source domain models in a hierarchical manner while protecting data privacy. The model can learn common features across domains, thereby realizing semantic information sharing of multi-source data, while avoiding direct sharing of data and ensuring the security of data privacy.

[0026] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of hierarchical model parameter fusion, and the specific process includes: determining the total number of source domains and the weight of each source domain based on source domain data; obtaining the parameters of each source domain model at the target network level respectively; performing weighted average calculation based on the total number of source domains, source domain weights, and parameters to obtain the fused model parameters of the target network level.

[0027] Based on the independently trained source domain model, in this embodiment, the computer device can fuse the model parameters in a hierarchical manner. Specifically, the model parameters are weighted averaged at different network levels to aggregate semantic information from different source domains. The mathematical expression of hierarchical parameter fusion can be described as: ;in, represents the fused l-th layer model parameters, is the parameter of the h-th source domain model at layer l, W h is the weight corresponding to the h-th source domain, and H is the total number of source domains.

[0028] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of semantic similarity measurement and weight allocation. The specific process includes: determining a reference model, and calculating a semantic similarity measurement between the reference model and the source domain data at the target network level; obtaining a similarity control coefficient, and calculating a weight allocation strategy based on the similarity control coefficient and the semantic similarity measurement; allocating model weights according to the weight allocation strategy, and adjusting the connection weights between different network levels, introducing differential privacy technology, and obtaining a fused source domain model.

[0029] Specifically, in order to effectively fuse the model parameters of different source domains, a semantic similarity metric can be introduced to determine the weight of each source domain in the semantic aggregation process. The weight distribution is based on the semantic similarity of the source domain model at a certain layer. The source domain with higher similarity has a greater weight in the aggregation process. The weight distribution strategy can be expressed as: ; Among them, S h is the semantic similarity measure between the h-th source domain and the central model (or reference model) at the l-th layer, and γ is the coefficient that controls the influence of similarity.

[0030] In order to ensure that semantic information can be effectively aggregated at different levels, in this embodiment, the optimization of cross-layer information flow is also considered. By adjusting the connection weights between different layers, the model's ability to learn deep semantic information can be enhanced. The optimization of cross-layer information flow can be expressed as: ;in, From the first layer to the The information flow of the layer, From the first layer to the The connection weight matrix of the layer is the parameter of the h-th source domain model at layer l, W h is the weight corresponding to the h-th source domain.

[0031] In the process of model parameter fusion involving multiple sources, fusion can be performed by not sharing the original data but only sharing the model parameters, which effectively protects data privacy. In addition, by introducing differential privacy technology, privacy protection can be further enhanced in the process of model parameter aggregation: ;in, is the model parameter that takes privacy protection into account during the fusion process, is the privacy budget of differential privacy, is the noise sampled from a Gaussian distribution, Represents the lth layer model parameters after fusion.

[0032] Step 206, using the attention mechanism to perform cross-level alignment and calibration operations on the aggregated semantic features to obtain calibrated optimized features.

[0033] In order to solve the semantic misalignment problem caused by differences in source domain data distribution, in this embodiment, the computer device can introduce a cross-layer semantic calibration mechanism, and use the attention mechanism to accurately align the semantic features of the fused model, thereby enhancing the model's generalization ability for different domains.

[0034] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of feature alignment and calibration operations, and the specific process includes: determining the feature dimensions of the aggregated semantic features, obtaining the attention weights of the source domain data on the feature dimensions, and determining the mapping function corresponding to the feature dimensions; calculating the attention-weighted feature representation based on the fused model parameters, attention weights, and mapping functions; and performing cross-layer feature alignment on the attention-weighted feature representation to obtain aligned features.

[0035] Specifically, in the multi-source domain learning process, data from different source domains may show different distribution patterns in the feature space. In order to capture these distribution patterns, we first need to learn the semantic feature representation of each source domain data. The formula for obtaining the feature representation can be expressed as: ;in, represents the semantic feature representation of h source domains at layer l, is the model parameter of the lth layer, and is the feature mapping function.

[0036] In order to accurately align the semantic features of different source domains, an attention mechanism can be introduced. The attention mechanism can focus on the most task-relevant part of the feature representation, thereby improving the model's sensitivity to key information. The attention-weighted feature representation can be expressed as: ;in, is the attention-weighted feature representation, is the dimension of the l-th layer feature, It is The attention weight of the source domain on the i-th feature dimension, is the mapping function of the i-th feature dimension, are the model parameters of the lth layer.

[0037] Through the feature representation weighted by the attention mechanism, cross-layer feature alignment is further performed to ensure that the semantic information learned by different source domain models at different levels can be coordinated with each other. The goal of feature alignment is to minimize the differences between feature representations of different source domains, which can be achieved by optimizing the problem: ;in, are model parameters, is the semantic feature representation of the source domain at the lth layer as a reference, are the model parameters of the lth layer.

[0038] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of domain invariance enhancement, and the specific process includes: determining the total number of network layers, calculating the feature average of all semantic features of the target network layer; and calculating the domain invariance loss based on the total number of network layers and the feature average.

[0039] Specifically, in order to enhance the domain invariance of the model, in this embodiment, a special loss function is set, which aims to minimize the distribution difference between feature representations of different source domains. The domain invariance loss can be expressed as: ; where L is the total number of network layers, is the attention-weighted feature representation, It is the average value of all source domain feature representations in layer l. This enables the model to achieve accurate alignment of semantic features between different source domains and enhances the model’s generalization ability to different domains.

[0040] Step 208 , based on the calibration optimization feature, the performance of the fused source domain model is evaluated, and the strength of the semantic calibration is adjusted according to the performance evaluation result to obtain the target source domain model.

[0041] Computer equipment can adaptively adjust the strength of semantic calibration based on the model's performance on the target task. This dynamic adjustment mechanism enables the model to adapt to different target domains more flexibly and further improve the generalization performance of the model.

[0042] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a process of evaluating the performance of the model. The specific process includes: determining a validation set on the target domain, inputting the validation set on the target domain into the fused source domain model, and obtaining the current performance of the fused source domain model; comparing the current performance with the reference performance to obtain a comparison result; and calculating the dynamic calibration strength based on the comparison result to obtain the target source domain model.

[0043] Specifically, in this embodiment, the computer device needs to evaluate the performance of the model on the target task, which can be achieved by testing the validation set on the target domain. The mathematical expression for the target task performance evaluation is: ;in, represents the performance of the model on the target task, is the validation set on the target domain, is the indicator function, when the model's prediction When it is equal to the true label y, the function value is 1, otherwise it is 0.

[0044] After obtaining the performance of the model on the target task, we can further analyze the difference between the model performance and the ideal performance of the target task. The ideal performance can be set based on historical data or expert knowledge, and the expression is: ;in, Indicates performance differences, , is the maximum value of all model performances based on historical data manually set by combining domain knowledge and experimental experience, and an additional expected gain value is added .

[0045] Then, the computer device can adaptively adjust the strength of semantic calibration according to the results of the performance difference analysis. The adjustment mechanism can be based on a simple linear mapping or a more complex nonlinear function. The mathematical expression of dynamic calibration strength adjustment is: ;in, is the dynamically adjusted semantic calibration strength, and f is the mapping function.

[0046] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, which may also include a dynamic calibration process, and the specific process includes: applying the dynamic calibration strength to the source domain model and obtaining the domain invariance loss; calculating the weighted calibration loss according to the dynamic calibration strength and the domain invariance loss; applying the weighted calibration loss to the source domain model to optimize the target source domain model.

[0047] Specifically, the computer device can apply the dynamically adjusted semantic calibration strength to the model training process, and further optimize the model through the weighted calibration loss, which is expressed as: ;in, is the dynamically adjusted semantic calibration strength, is the weighted calibration loss, is the domain invariance loss.

[0048] Finally, the computer device can combine the weighted calibration loss with other training losses to perform end-to-end model optimization. The expression of end-to-end model optimization is updated as: ;in, represents the cross entropy loss function; represents the contrast loss; α, β are the weights used to weight the calibration loss, is the weighted calibration loss. The strength of semantic calibration is automatically adjusted according to the model’s performance on the target task, making the model more flexible to adapt to different target domains.

[0049] A domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration provided in this application effectively extracts and fuses key semantic information from multiple source domains. In cross-domain image recognition tasks, especially on PACS datasets, it can achieve higher accuracy than the existing method EISNet, with an average accuracy improvement of about 3.4%. This result proves the improvement of the model's generalization ability, especially when processing data from different source domains; independently training models on different source domains and performing semantic aggregation in a data-independent manner avoids direct sharing of raw data. This method not only protects data privacy, but also avoids the risks caused by data leakage, and complies with the requirements of current data protection regulations; optimizes the model training process and reduces the demand for computing resources; can adaptively adjust the calibration intensity according to the performance of the model on the target task, making the model training process more efficient. Compared with traditional fixed strategies, on the Digits-DG dataset, it can converge faster and with fewer training iterations. The proposed method can achieve better performance within a certain number of tasks; the semantic features of the fused model are accurately aligned through the attention mechanism, which effectively solves the semantic misalignment problem caused by the difference in the distribution of source domain data; on the PACS and VLCS datasets, it can better adapt to the distribution of data in different source domains, improve the generalization ability of the model on diversified data, and improve the average accuracy of the existing method EISNet by about 4.1%; it is not only suitable for the domain generalization problem of a single task, but can also be extended to multi-task learning scenarios, and can achieve effective knowledge transfer between different tasks; through hierarchical feature fusion and calibration, the interpretability of the model decision process is improved. This transparency enables users and technicians to better understand and trust the prediction results of the model, especially in application scenarios with high requirements for model interpretability, such as medical image analysis, industrial manufacturing, and financial analysis.

[0050] The present application provides a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration. While protecting data privacy, the model's generalization ability for unseen target domains is significantly improved through multi-source hierarchical semantic fusion and cross-layer calibration. First, the model is trained independently on each source domain to obtain local semantic information; then, a data-independent semantic aggregation strategy is adopted to hierarchically fuse the parameters of each source domain model to achieve semantic information sharing of multi-source data, while avoiding direct sharing of data and effectively protecting data privacy; further, in order to solve the semantic dislocation problem caused by differences in source domain data distribution, a cross-layer semantic calibration mechanism is introduced, and the attention mechanism is used to accurately align the semantic features of the fused model, thereby enhancing the domain invariance of the model. In addition, a dynamically adjusted semantic calibration strategy is proposed, which adaptively adjusts the calibration strength according to the performance of the model on the target task to further improve the generalization ability of the model. Through these innovative technologies, the generalization ability of the model in multi-source domains can be effectively improved, while ensuring the security of data privacy, which is suitable for application scenarios such as medical image analysis, cross-modal data fusion, and privacy-preserving machine learning.

[0051] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0052] In one embodiment, Figure 3 As shown, a domain generalization system based on multi-source hierarchical semantic fusion and cross-layer calibration is provided, including: a local semantic acquisition module 310, a semantic aggregation module 320, a cross-layer semantic calibration module 330 and a dynamic semantic calibration strategy module 340, wherein: The local semantic acquisition module 310 is used to acquire source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of the local data; The semantic aggregation module 320 is used to perform weighted averaging of the parameters of each source domain model at different network levels, and introduce semantic similarity measurement to distribute model weights, so as to obtain a fused source domain model, and aggregate various semantic features through the fused source domain model; A cross-layer semantic calibration module 330 is used to perform a cross-layer alignment calibration operation on the aggregated semantic features using an attention mechanism to obtain calibrated optimized features; The dynamic semantic calibration strategy module 340 is used to perform performance evaluation on the fused source domain model based on the calibration optimization feature, and adjust the strength of the semantic calibration according to the performance evaluation result to obtain the target source domain model.

[0053] Each module cooperates with each other to form a complete domain generalization technical solution. The local semantic acquisition module 310 provides the initial semantic information for the entire system; the semantic aggregation module 320 and the cross-layer semantic calibration module 330 work together to achieve the deep learning and generalization of the model for multi-source data; finally, the cross-layer semantic calibration module 330 ensures the adaptability of the model to the target domain. It not only effectively utilizes multi-source data, but also protects data privacy, providing an innovative solution to the domain generalization problem.

[0054] In one embodiment, the local semantic acquisition module 310 is also used to obtain each source domain data and use each source domain data for model training; extract the feature representations in each source domain data respectively, minimize the classification loss on the source domain based on each feature representation, and complete the training of the source domain model.

[0055] The local semantic acquisition module 310 is responsible for independently training models on their respective source domains to extract and learn the semantic information of local data. Figure 4 As shown, the local semantic acquisition module 310 may include: data input, local semantic acquisition module, feature representation, loss function, local semantic information, among which: Data input: source domain data input, providing raw data for model training; Local semantic acquisition module: responsible for model training and semantic information extraction; Feature representation: high-level feature representation extracted by the model, containing rich semantic information; Loss function: loss function used to optimize the model, including cross entropy loss, contrast loss and L2 regularization loss; Local semantic information: The semantic information output by the module provides a basis for subsequent aggregation.

[0056] The workflow includes: data enters the local semantic acquisition module through input; in the local semantic acquisition module, data is used to train the model and extract feature representation; the feature representation is optimized through the loss function to ensure that the model can effectively capture the semantic information of the data; finally, the local semantic acquisition module outputs local semantic information to provide input for the data-independent semantic aggregation module.

[0057] In one embodiment, the semantic aggregation module 320 is also used to determine the total number of source domains and the weight of each source domain based on the source domain data; obtain the parameters of each source domain model at the target network level respectively; perform weighted average calculation based on the total number of source domains, source domain weights, and parameters to obtain the fused model parameters of the target network level.

[0058] In one embodiment, the semantic aggregation module 320 is also used to determine a reference model and calculate a semantic similarity measure between the reference model and the source domain data at the target network level; obtain a similarity control coefficient, and calculate a weight allocation strategy based on the similarity control coefficient and the semantic similarity measure; perform model weight allocation according to the weight allocation strategy, and adjust the connection weights between different network levels, introduce differential privacy technology, and obtain a fused source domain model.

[0059] The semantic aggregation module 320 is responsible for fusing the parameters of different source domain models in a hierarchical manner while protecting data privacy. Figure 5 As shown, the semantic aggregation module 320 may include: local semantic information, data-independent semantic aggregation module, fusion model parameters, information flow, and privacy protection parameters, among which: Local semantic information: output by the local semantic acquisition module, representing the semantic information of each source domain; Data-independent semantic aggregation module: responsible for hierarchical fusion of parameters of different source domain models to achieve semantic information sharing; Fusion model parameters: Fusion model parameters obtained by weighted averaging of different source domain model parameters; Information flow: represents the result of cross-layer information flow optimization, which enhances the model's ability to learn deep semantic information; Privacy protection parameters: Parameters that take privacy protection into account during the fusion process.

[0060] The workflow includes: local semantic information is sent to the data-independent semantic aggregation module; in the data-independent semantic aggregation module, model parameters of different source domains are fused in a hierarchical manner to generate fused model parameters; through information flow, the module optimizes cross-layer information flow to enhance the model's learning ability for deep semantic information; privacy protection parameters are introduced to ensure data privacy during the model parameter fusion process; finally, the semantic aggregation module 320 outputs the updated fused model parameters for subsequent processing by the cross-layer semantic calibration module 330.

[0061] In one embodiment, the cross-layer semantic calibration module 330 is also used to determine the feature dimensions of the aggregated semantic features, obtain the attention weights of the source domain data on the feature dimensions, and determine the mapping function corresponding to the feature dimensions; calculate the attention-weighted feature representation based on the fused model parameters, attention weights, and mapping functions; perform cross-layer feature alignment on the attention-weighted feature representation to obtain aligned features.

[0062] In one embodiment, the cross-layer semantic calibration module 330 is also used to determine the total number of network levels, calculate the feature average of all semantic features of the target network level; and calculate the domain invariance loss based on the total number of network levels and the feature average.

[0063] The cross-layer semantic alignment module 330 is the key to solving the semantic misalignment problem caused by the difference in source domain data distribution. By introducing a sophisticated semantic feature alignment mechanism, the model's generalization ability for different source domain data is enhanced. Figure 6 As shown, the cross-layer semantic calibration module 330 may include: fusion model parameters, cross-layer semantic calibration, semantic feature representation, attention weight, aligned feature representation, domain invariance loss, among which: Fusion model parameters: output by the data-independent semantic aggregation module, representing the fused model parameters; Cross-layer semantic alignment: responsible for performing representation learning and alignment of semantic features; Semantic feature representation: represents the learned semantic features; Attention weight: The weight obtained through the attention mechanism is used for weighted feature representation; Aligned feature representation: feature representation optimized by cross-layer alignment; Domain Invariance Loss: A loss function used to enforce domain invariance of the model.

[0064] The workflow includes: fusing model parameters into cross-layer semantic calibration; in cross-layer semantic calibration, firstly learning semantic feature representation to obtain semantic feature representation; using the attention mechanism, calculating the attention weight and weighting the feature representation; the weighted semantic feature representation is optimized through cross-layer alignment to obtain aligned feature representation; by minimizing the domain invariance loss, the model's generalization ability for data from different source domains is enhanced; the output of the cross-layer semantic calibration module is the calibrated and optimized feature representation, which is used to further improve the model performance.

[0065] In one embodiment, the dynamic semantic calibration strategy module 340 is also used to determine a verification set on the target domain, input the verification set on the target domain into the fused source domain model, and obtain the current performance of the fused source domain model; compare the current performance with the reference performance to obtain a comparison result; calculate the dynamic calibration strength based on the comparison result to obtain the target source domain model.

[0066] In one embodiment, the dynamic semantic calibration strategy module 340 is also used to apply the dynamic calibration strength to the source domain model and obtain the domain invariance loss; calculate the weighted calibration loss according to the dynamic calibration strength and the domain invariance loss; apply the weighted calibration loss to the source domain model to optimize the target source domain model.

[0067] The dynamic semantic calibration strategy module 340 is used to further improve the generalization ability of the model and can adaptively adjust the strength of semantic calibration according to the performance of the model on the target task. Figure 7 As shown, the dynamic semantic calibration strategy module 340 may include: calibrated feature representation, dynamic semantic calibration strategy, performance evaluation, performance difference analysis, dynamic calibration strength, weighted calibration loss, and optimized model, wherein: Calibrated feature representation: Output by the cross-layer semantic calibration module, representing the semantically calibrated feature representation; Dynamic semantic calibration strategy module: responsible for dynamically adjusting the semantic calibration strategy based on model performance; Performance evaluation: Evaluate the performance of the model on the target task; Performance gap analysis: Analyze the gap between model performance and ideal performance; Dynamic calibration strength: Calibration strength that is dynamically adjusted based on performance differences; Weighted calibration loss: Apply dynamic calibration strength to the calibration loss; Optimized model: The model is optimized end-to-end.

[0068] The workflow includes: the calibrated feature representation is fed into the dynamic semantic calibration strategy; the dynamic semantic calibration strategy first performs a performance evaluation to determine the current performance level of the model; through performance difference analysis, the module determines the gap between the model performance and the ideal performance; based on the results of the performance difference analysis, the module calculates the dynamic calibration strength to adjust the strength of the semantic calibration; the dynamic calibration strength is used to weight the calibration loss to further optimize the model; through an end-to-end optimization process, the optimized model is finally output, which has better generalization ability.

[0069] In one embodiment, the end-to-end model optimization process can ensure that the model can achieve good performance on all source domains while maintaining the generalization ability of the model. Figure 8 As shown, the components include: optimized model, end-to-end model optimization, performance evaluation, and optimization results, where: Optimized model: Output by the dynamic semantic calibration strategy module, representing the dynamically calibrated model; End-to-end model optimization: responsible for the final optimization of the entire model to ensure generalization capabilities; Performance evaluation: Conduct a comprehensive performance evaluation of the optimized model; Optimization result: The final result of model optimization, used for the target task.

[0070] In this embodiment, the optimized model enters the end-to-end model optimization process; the model first undergoes performance evaluation to comprehensively evaluate the performance of the model; based on the results of the performance evaluation, the end-to-end model optimization process makes necessary adjustments and optimizations to the model; finally, the end-to-end model optimization outputs the optimization result, which is the final achievement of the model optimization and has better generalization performance.

[0071] In one embodiment, a signaling structure diagram of a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is provided as follows: Fig. 9 As shown in the figure, the signaling and structure transmitted between different modules during model training are shown, including: Local semantic learning signaling: sent by the local semantic acquisition module, containing semantic feature information specific to the source domain; Parameter aggregation signaling: sent by the semantic aggregation module, containing parameter information aggregated from multiple source domain models; Semantic calibration signaling: sent by the cross-layer semantic calibration module, containing calibration parameters for aligning semantic features of different source domains; Performance evaluation signaling: Sent by the dynamic semantic calibration strategy module, containing the performance evaluation results of the model on the target task; Dynamic adjustment signaling: Based on the performance evaluation results, the dynamic semantic calibration strategy module sends a signaling to adjust the semantic calibration strength.

[0072] exist Fig. 9 In the figure, the interaction between signaling is represented by arrows, and the labels on the arrows describe the direction and purpose of the signaling. For example, local semantic learning signaling provides information to parameter aggregation signaling; parameter aggregation signaling triggers semantic calibration signaling; semantic calibration signaling requires performance evaluation signaling to verify the calibration effect; performance evaluation signaling determines whether to send dynamic adjustment signaling to optimize model performance.

[0073] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is applied to an intelligent recommendation system of multi-source data fusion to provide more personalized and accurate user recommendation services. Fig.10 The specific implementation process is as follows: Multi-source data collection: collect user data from different sources, such as purchase history, browsing behavior, user feedback, etc. Local semantic analysis: Analyze each data source independently to extract local semantic features of user behavior; Data-independent feature aggregation: local features from different data sources are integrated to form a comprehensive representation of user behavior; Cross-source feature calibration: The fused features are calibrated through the attention mechanism to enhance the consistency and complementarity between different data sources; Recommendation model training and optimization: Use calibrated features to train the recommendation model and continuously optimize the model to provide more accurate recommendations; Personalized recommendation output: The model outputs personalized recommendation results based on the user's comprehensive behavioral characteristics; User feedback collection and evaluation: Collect user feedback on recommendation results and evaluate the performance of the recommendation system; Performance feedback and model iteration: Iteratively optimize the recommendation model based on user feedback and performance evaluation results.

[0074] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is applied to a cross-modal fusion diagnosis system of medical images to improve the accuracy and efficiency of disease diagnosis. Fig.11 The specific implementation process is as follows: Image data acquisition: In a medical environment, medical image data of patients is collected through different imaging devices (such as CT and MRI); Local feature extraction: The model is trained independently on the dataset of each modality to extract the local features of the image and capture the semantic information specific to the modality. Data-independent feature aggregation: The features of images of different modalities are integrated in a hierarchical manner to achieve semantic information sharing of multimodal data; Cross-modal feature calibration: Use the attention mechanism to calibrate the fused features to ensure the consistency and complementarity of features between different modalities; Diagnostic model training and optimization: Based on the fused and calibrated features, the diagnostic model is trained and optimized to improve the accuracy of the model in disease diagnosis; Diagnosis output and evaluation: The model outputs the final diagnosis result, which is evaluated and interpreted by the doctor; Performance feedback and model iteration: Based on the evaluation feedback of the diagnosis results, the model is iteratively optimized to continuously improve the diagnosis performance.

[0075] In one embodiment, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is applied to surface defect detection of industrial products to improve the accuracy and efficiency of detection. Fig.12 The specific implementation process is as follows: Image and point cloud data acquisition: On the production line, high-resolution cameras and 3D scanners are used to obtain RGB images and point cloud data of products; Data preprocessing: De-noise and standardize the collected data to eliminate the inconsistency between data collected by different devices; Feature extraction: Use deep learning models to extract color and texture features from RGB images and shape features from point clouds; Feature fusion: RGB features and point cloud features are combined through a specific fusion strategy to form a comprehensive feature representation; Anomaly detection model training: The fused features are used to train the anomaly detection model, which can identify the feature distribution of normal products. Online anomaly detection: During the production process, products are scanned and analyzed in real time, and the model predicts whether the product contains defects; Defect location and assessment: For products that the model determines to be abnormal, further locate the defect area and assess the severity of the defect; Decision-making and feedback: Based on the evaluation results, products are automatically or manually rejected or repaired, and feedback is provided to the production control system; Model optimization: Regularly optimize the model based on test results and feedback to adapt to changes in the production process.

[0076] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0077] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, steps of a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration are implemented.

[0079] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration are implemented.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0081] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration, characterized in that: The method comprises: Acquire source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of local data; The parameters of each source domain model are weighted averaged at different network levels, and a semantic similarity metric is introduced to distribute model weights to obtain a fused source domain model, and each semantic feature is aggregated through the fused source domain model; The attention mechanism is used to perform cross-level alignment and calibration operations on the aggregated semantic features to obtain calibrated and optimized features. Based on the calibration optimization feature, the performance of the fused source domain model is evaluated, and the strength of the semantic calibration is adjusted according to the performance evaluation result to obtain a target source domain model.

2. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 1, characterized in that: Obtain source domain data and independently train each source domain to obtain a source domain model, including: Acquire various source domain data, and use the various source domain data to perform model training; Feature representations in each of the source domain data are extracted respectively, and the classification loss on the source domain is minimized based on each of the feature representations to complete the training of the source domain model.

3. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 1 is characterized in that: The parameters of each source domain model are weighted averaged at different network levels, including: Determine the total number of source domains and the weight of each source domain based on the source domain data; Respectively obtaining parameters of each of the source domain models at the target network level; A weighted average calculation is performed according to the total number of source domains, source domain weights, and parameters to obtain fused model parameters of the target network level.

4. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 3 is characterized in that: The semantic similarity metric is introduced to distribute the model weights and obtain the fused source domain model, including: Determining a reference model, and calculating a semantic similarity measure between the reference model and the source domain data at the target network level; Obtaining a similarity control coefficient, and calculating a weight allocation strategy according to the similarity control coefficient and a semantic similarity metric; The model weights are distributed according to the weight distribution strategy, the connection weights between different network layers are adjusted, and the differential privacy technology is introduced to obtain the fused source domain model.

5. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 3 is characterized in that: The attention mechanism is used to perform cross-level alignment and calibration operations on the aggregated semantic features, including: Determine a feature dimension of the aggregated semantic feature, obtain an attention weight of the source domain data on the feature dimension, and determine a mapping function corresponding to the feature dimension; Calculating an attention-weighted feature representation according to the fused model parameters, attention weights, and mapping functions; Perform cross-layer feature alignment on the attention-weighted feature representation to obtain aligned features.

6. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 3, characterized in that: The method further comprises: Determine the total number of network levels, and calculate the feature average of all semantic features of the target network level; The domain invariance loss is calculated based on the total number of network layers and the feature average.

7. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 6, characterized in that: Based on the calibration optimization feature, the performance of the fused source domain model is evaluated, and the strength of the semantic calibration is adjusted according to the performance evaluation result to obtain a target source domain model, including: Determine a validation set on the target domain, input the validation set on the target domain into the fused source domain model, and obtain the current performance of the fused source domain model; Comparing the current performance with the reference performance to obtain a comparison result; The dynamic calibration strength is calculated according to the comparison result to obtain the target source domain model.

8. The domain generalization method based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 7, characterized in that: The method further comprises: Applying the dynamic calibration strength to a source domain model and obtaining the domain invariance loss; Calculating a weighted calibration loss according to the dynamic calibration strength and the domain invariance loss; The weighted calibration loss is applied to the source domain model and the target source domain model is obtained by optimization.

9. A domain generalization system based on multi-source hierarchical semantic fusion and cross-layer calibration, characterized in that: The system comprises: A local semantic acquisition module is used to acquire source domain data, independently train each source domain to obtain a source domain model, and optimize the corresponding source domain data through the source domain model to extract and learn the semantic features of local data; A semantic aggregation module, used to perform weighted averaging of the parameters of each source domain model at different network levels, and introduce semantic similarity measurement to distribute model weights, so as to obtain a fused source domain model, and aggregate various semantic features through the fused source domain model; The cross-layer semantic calibration module is used to use the attention mechanism to perform cross-layer alignment and calibration operations on the aggregated semantic features to obtain calibrated optimized features; The dynamic semantic calibration strategy module is used to perform performance evaluation on the fused source domain model based on the calibration optimization feature, and adjust the strength of the semantic calibration according to the performance evaluation result to obtain a target source domain model.

10. The domain generalization system based on multi-source hierarchical semantic fusion and cross-layer calibration according to claim 9, characterized in that: The local semantic acquisition module is also used to: obtain each source domain data, use each source domain data for model training; extract the feature representations in each source domain data respectively, minimize the classification loss on the source domain based on each feature representation, and complete the training of the source domain model.

Images