Intelligent algorithm precision training optimization method based on multi-modal fusion

By constructing a multimodal dataset and a reinforcement learning agent, the optimal fusion strategy is automatically explored, solving the problem of low adaptation efficiency of multimodal fusion strategies in flexible manufacturing scenarios, and achieving rapid adaptation and efficient detection.

CN123070232APending Publication Date: 2026-11-10上海研汐科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611431885.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multimodal fusion strategies are difficult to adapt quickly to the modal data distribution and defect types of different product lines in flexible manufacturing scenarios, resulting in slow iteration speed of the detection system, high production line downtime costs, and low efficiency of existing neural architecture search solutions, which are prone to getting stuck in local optima and cannot be dynamically adjusted.

Method used

A multimodal dataset containing labeled training subsets and unlabeled pre-training subsets is constructed. By combining reinforcement learning agents with evolutionary algorithms, the optimal fusion strategy is automatically explored. A performance-driven step learning rate strategy is used for end-to-end training, supporting two-stage incremental transfer, and realizing automated and customized optimization of the multimodal fusion model.

Benefits of technology

It enables rapid adaptation of multimodal fusion models, shortens the development cycle of fusion strategies, improves detection accuracy and model performance, reduces deployment costs, and supports rapid scene migration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070232A_ABST
    Figure CN123070232A_ABST
Patent Text Reader

Abstract

The application discloses a multi-modal fusion-based intelligent algorithm precise training optimization method, and relates to the technical field of artificial intelligence multi-modal model training optimization; the application constructs a multi-modal data set containing a labeled training subset and an unlabeled pre-training subset, completes double-stage baseline initialization of each modal feature extraction network, encodes the number of hierarchical levels, feature dimensions, interaction modes and transmission paths of the fusion strategy into a standardized optimization space, adopts a reinforcement learning agent combined with an evolutionary algorithm for auxiliary iteration, automatically explores an optimal fusion strategy, and after hierarchical topological decoding, adopts a performance-driven stepwise learning rate strategy to complete end-to-end precise training, simultaneously supports double-stage incremental migration, and realizes automatic and customized optimization of the multi-modal fusion model in different task scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence multimodal model training and optimization technology, specifically to a method for precise training and optimization of intelligent algorithms based on multimodal fusion. Background Technology

[0002] In the surface quality inspection of parts in discrete manufacturing enterprises, multimodal inspection solutions that integrate visible light images, infrared thermal imaging, and vibration sensing signals are commonly used to cover various types of defects such as cracks, deformation, and internal stress. However, as product models on production lines iterate and inspection accuracy requirements increase, the distribution of modal data and the proportion of defect types differ significantly across product lines, necessitating targeted adjustments to the multimodal model fusion strategy. Furthermore, the adaptability efficiency of the fusion strategy directly determines the iteration speed of the inspection system and the downtime costs of the production line, a problem particularly prominent in flexible manufacturing scenarios with frequent product changes.

[0003] Existing multimodal fusion strategies largely rely on manual design and optimization by algorithm engineers. For new production line scenarios, repeated testing of fusion levels, interaction methods, and feature dimension parameters is necessary, with debugging cycles reaching several weeks and heavily dependent on personnel experience. It is difficult to find a globally optimal solution simultaneously across the three engineering constraints of detection accuracy, model parameter count, and inference latency. While some solutions employing Neural Architecture Search (NAS) achieve automation, their search spaces are often highly dispersed and enumerable, resulting in low optimization efficiency, susceptibility to local optima, and a disconnect between the search strategy and subsequent training processes, making it impossible to dynamically adjust the search direction based on validation set feedback.

[0004] Furthermore, existing solutions lack a unified framework for handling specific needs common in industrial scenarios, such as adapting temporal modal sampling parameters and adjusting small target detection weights, resulting in significant manual intervention required for algorithm implementation. Therefore, a precise training method is needed that can automatically optimize multimodal fusion strategies, balance accuracy and engineering constraints, and support rapid scenario migration. This method aims to overcome the limitations of manual design based on experience and meet the production needs of enterprises for rapid product transitions. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a precise training and optimization method for intelligent algorithms based on multimodal fusion. It can complete the two-stage baseline initialization of each modality feature extraction network by constructing a multimodal dataset containing labeled training subsets and unlabeled pre-training subsets. It encodes the number of layers, feature dimensions, interaction methods and transmission paths of the fusion strategy into a standardized optimization space. It uses reinforcement learning agents combined with evolutionary algorithms to assist iteration and automatically explore the optimal fusion strategy. After hierarchical topology decoding, it uses a performance-driven step learning rate strategy to complete end-to-end precise training. At the same time, it supports two-stage incremental transfer and realizes automated and customized optimization of multimodal fusion models under different task scenarios.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for precise training and optimization of intelligent algorithms based on multimodal fusion, comprising the following components: Step 1: Construct a multimodal dataset corresponding to the target task. The dataset includes a labeled training subset and an unlabeled pre-training subset. Build an independent feature extraction network for each modality, extract high-dimensional feature representations of the data from each modality, and complete the initialization of the baseline model. Step 2: Encode the number of multimodal fusion layers, feature dimensions of each layer, modal interaction methods, and feature propagation paths into a discretized action space. Encode the validation set performance, number of parameters, and inference latency corresponding to the current fusion structure into a continuous state space. Complete the standardized mapping of the optimization space. The feature propagation path is the configuration parameter of the cross-layer residual connection between fusion layers. Step 3: Construct a reinforcement learning optimization agent using a proximal policy optimization algorithm. Construct a weighted combined reward function that includes core task performance terms and model constraint penalty terms to drive the agent to explore a fusion strategy that meets both performance and constraint requirements. Step 4: Initialize an evolutionary population containing multiple fusion strategy individuals. Generate an initial set of high-quality strategies as a pre-training experience pool for the agent through selection, crossover, and mutation operations. After every 15 rounds of reinforcement learning iterations, perform an evolutionary population update operation to expand the search space. Step 5: When the optimization process reaches the preset iteration rounds or performance threshold, select the current optimal action sequence and decode it into the corresponding multimodal fusion network structure. Then, concatenate the fusion network with the feature extraction networks of each modality and use a step-by-step learning rate decay strategy to complete end-to-end training. Step 6: When the target task or data distribution changes, the agent is fine-tuned based on the trained reinforcement learning agent network parameters and historical policy experience pool to generate a fusion policy adapted to the new scenario, thus completing the incremental update and cross-scenario transfer of the model.

[0007] Further, the baseline model initialization in step one is implemented according to the following process: Select samples from an unlabeled pre-training subset in the training set, accounting for 10% of the total samples in the training set. Use InfoNCE cross-modal contrastive learning loss to constrain the bottom-level parameters of each modality feature extraction network. Bring the different modal features corresponding to the same multimodal sample closer in the semantic space and push the modal features of different samples further apart. Set the temperature coefficient to 0.07. After completing 20 rounds of pre-training, freeze the bottom-level parameters of each modality feature extraction network. The bottom-level parameters refer to the trainable parameters of the first 80% of the network layers closest to the input side. Only the top-level parameters are retained for subsequent supervised fine-tuning. The top-level parameters refer to the parameters of the last 20% of the fully connected layers closest to the output side. The fine-tuning process uses samples from the labeled training subset. After each round of training, verify the accuracy of the single-modal task. When the decrease in accuracy compared with the pre-trained single-modal accuracy does not exceed 1%, the baseline model initialization is deemed qualified.

[0008] Furthermore, the optimization space normalization mapping in step two employs a discrete-continuous dual-domain embedding coding method, and the continuous action embedding vector is calculated using the following formula: ,in Embed the output continuous action vector with a dimension of 32; The discrete action vector is obtained by concatenating four types of discrete parameters—number of fusion layers, feature dimension of each layer, modal interaction method, and feature transmission path—after one-hot encoding. The learnable embedding matrix has a size of 32×D, where D is the total dimension of the discrete action vectors. The initial value is obtained by performing principal component analysis on the baseline features of each modality in the training set, concatenating the first 32-dimensional principal component vectors and then transposing them. The prior constraint weight is set to 0.2, which is determined based on the statistical mean of the intermodal feature differences from 10 publicly available multimodal benchmark datasets. The modal statistical prior vector is obtained by concatenating the median of each modal feature dimension and the mean of the Shannon entropy of the feature channels after min-max normalization to the [0,1] interval.

[0009] Furthermore, the weighted combination reward function in step three employs an adaptive dynamic weighting mechanism, calculating the total reward value using the following formula: ,in This represents the total reward value for the current iteration step. This is a normalized value of the performance metric of the target task on the validation set, with a value range of 0-1. This is the normalized value of the model parameters, calculated with the preset maximum parameter threshold as the denominator, and the value range is 0-1; This is the normalized value of the single-sample inference latency of the model, calculated with the preset maximum single-sample inference latency threshold as the denominator, and its value ranges from 0 to 1. , , For the first The dynamic weight coefficients corresponding to each iteration satisfy the following conditions: Early stage of iteration When using initial weights, The value is 0.8. and Each value is 0.1. When t≥5 iterations, the weights are dynamically adjusted based on the moving average convergence rate of the performance on the first 5 validation sets. When the convergence rate is greater than 0.5%, The value is 0.85. and When each value is 0.075, the convergence rate is between 0.2% and 0.5%. The value is 0.75. and When each value is 0.125, and the convergence rate is less than 0.2%, The value is 0.65. and Each value is 0.175.

[0010] Furthermore, the evolutionary population selection operation in step four employs a fitness calculation method weighted by modal complementarity, using the following formula to calculate the individual fitness value: ,in The fitness value of an individual using the fusion strategy; These are the original values ​​of the performance metrics of the model corresponding to this strategy on the validation set; The complementarity influence coefficient is set to 0.3, which is determined based on the statistical value of the contribution of modal complementarity information to performance in multimodal classification and detection tasks. The modal complementarity coefficient ranges from 0 to 1. The calculation logic is the ratio of the difference between mutual information and redundancy to the mutual information, taking a non-negative value. Mutual information is the average mutual information among all modal features, and redundancy is the mean of the Pearson correlation coefficients among all modal features. Higher mutual information and lower redundancy result in lower redundancy. The larger the value, the better.

[0011] Furthermore, the evolutionary crossover and mutation operations in step four are performed according to the following rules: After every 15 rounds of reinforcement learning iterations, an update operation is performed on the current population. In the selection phase, the fitness value is used as the sorting criterion, and the top 20% of individuals in the population are selected as high-quality parent individuals. In the crossover phase, a single-point crossover method is used, and an action dimension is randomly selected as the crossover site. All parameters after the crossover site of two high-quality parent individuals are swapped. The crossover probability is set to 0.7, and the number of offspring individuals generated is the same as the number of high-quality parent individuals. In the mutation phase, Gaussian mutation is applied to the newly generated offspring individuals. Three action dimensions are randomly selected as mutation sites, and the mutation amplitude is ±10% of the corresponding dimension value range. The mutation probability is set to 0.1. Finally, the high-quality parent individuals and the mutated offspring individuals are mixed to form a new population. The population size is consistent with the initial population size, and the high-quality strategies in the new population are synchronously stored in the reinforcement learning experience pool.

[0012] Furthermore, the optimal fusion strategy decoding in step five adopts a hierarchical topology mapping mechanism, performing mapping operations sequentially according to the hierarchical order. First, the number of fusion layers in the action sequence is mapped to the depth structure of the fusion network, with a one-to-one correspondence between the number of layers and the parameter values. Then, the feature dimension parameters of each layer are mapped to the number of output channels of the corresponding network layer. Next, the modal interaction mode parameters are mapped to the corresponding fusion operators. The concatenation parameters correspond to the channel dimension concatenation operator, the element-wise addition parameters correspond to the element-wise summation operator, and the attention parameters correspond to the single-head cross-attention operator. Finally, the feature transfer path parameters are mapped to the residual connection branches between layers. During the mapping process, the feature dimension matching of adjacent layers is automatically checked. When a dimension mismatch occurs, a 1×1 convolutional layer is inserted for dimension adaptation. Finally, a directly callable neural network computation graph is generated, which is directly connected to the output layer of each modality feature extraction network.

[0013] Furthermore, the end-to-end training in step five employs a performance-driven, tiered learning rate dynamic scheduling strategy, with the initial learning rate set to 2.5 × 10⁻⁻⁶. 5 The total training epochs were set to 40 epochs. The training process was divided into two phases: the first 10 epochs froze the low-level parameters of each modality feature extraction network and trained only the parameters of the fusion network; the next 30 epochs unfroze all network parameters for global fine-tuning. A validation set performance evaluation was performed every 8 epochs. When the performance improvement was less than 0.2% for two consecutive epochs, learning rate decay was triggered, with a decay rate of 0.6, and the cumulative decay count not exceeding 3 times. After reaching the decay cap, the current learning rate was maintained until the end of training. During training, the gradient clipping threshold was set to 1.0, and the L2 weight decay coefficient was set to 1 × 10⁻⁻⁶. 4 If the performance of the validation set declines for three consecutive rounds, the training process will be terminated early.

[0014] Furthermore, the incremental update and cross-scenario migration in step six adopt a two-stage migration calibration mechanism: In the first stage, the underlying policy network parameters of the reinforcement learning agent are frozen, and only the top-level decision head parameters are fine-tuned. Based on the validation set data of the new scenario, 15 rounds of iteration are performed to adapt to the performance evaluation criteria and modal feature distribution of the new scenario. In the second stage, the top 20% of the parameters in the underlying policy network are unfrozen, and 10 rounds of fine-tuning are performed in combination with the policy exploration experience of the new scenario. During the iteration process, the historical policy experience pool is updated, retaining 70% of the historical high-quality policy samples in the original experience pool and supplementing 30% of the new scenario policy samples. The mixture is used as the experience basis for subsequent iterations. When the number of labeled samples in the new scenario exceeds 60% of the number of labeled samples in the original scenario, the full optimization process is started to re-search for the optimal fusion strategy.

[0015] Furthermore, corresponding adaptation processing is adopted for different task scenarios. For temporal modal tasks that include voice and video, a temporal window length dimension is added to the action space, with a value range of 5-30 frames and a step size of 5 frames. During the training phase, training samples are generated by sliding window sampling of temporal modal data, with the window overlap rate set to 50%. Sine and cosine position encoding is added to the input layer of the corresponding temporal modality in the fusion network, and the encoding dimension is consistent with the feature dimension of the corresponding modality. For industrial multimodal detection tasks, affine transformation and bilinear interpolation are used to unify the spatial resolution for two-dimensional modalities such as images and infrared, and the alignment accuracy is controlled within 2px. For one-dimensional modalities such as vibration and soundprint, timestamp synchronization processing is used, and the time alignment accuracy is controlled within 1ms. During the training phase, small target samples are oversampled at a ratio of 3:1, and the recall rate of small targets accounts for no less than 30% of the weight of the performance reward item.

[0016] Compared with existing technologies, this intelligent algorithm-based precise training and optimization method based on multimodal fusion has the following beneficial effects: I. This method constructs a standardized fusion strategy optimization space through discrete-continuous dual-domain encoding. Combining the sequential decision-making ability of reinforcement learning with the global search ability of evolutionary algorithms, it can automatically explore the optimal fusion topology and parameter configuration that are suitable for the target task, replacing the traditional manual parameter tuning mode, significantly shortening the fusion strategy development cycle, and achieving a balance of multiple constraints such as task performance, model parameter quantity, and inference latency, thereby improving the global optimality of the fusion scheme.

[0017] Second, this method adopts a two-stage baseline initialization, hierarchical topology decoding and performance-driven training scheduling mechanism to ensure the convergence stability and feature utilization efficiency of the fusion model. At the same time, it supports two-stage incremental transfer and scenario adaptation, which can quickly adapt to the application requirements of different modal combinations and different task types, and reduce the cost and iteration difficulty of multimodal model deployment in industrial scenarios.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 The main flowchart is shown for the precise training and optimization method of intelligent algorithms based on multimodal fusion. Figure 2 A flowchart for the baseline model initialization process of a precise training and optimization method for intelligent algorithms based on multimodal fusion; Figure 3 A flowchart illustrating the reinforcement learning and evolutionary population joint optimization process for a precise training and optimization method of intelligent algorithms based on multimodal fusion. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] In industrial multimodal defect detection scenarios, equipment simultaneously acquires visible light images of the workpiece, two-dimensional modal data from infrared thermal imaging, and one-dimensional temporal modal data from equipment vibration to identify surface cracks, internal thermal damage, and vibration-induced latent defects. This scenario presents challenges such as significant modal heterogeneity, inconsistent modal spatial dimensions, a low proportion of small defect samples, and strong complementarity between different modal information. Traditional fixed-structure multimodal fusion networks cannot adaptively match the current detection task, often resulting in insufficient detection accuracy, excessive model parameters, and high inference latency, failing to meet the real-time detection requirements of production lines. This embodiment is applied to this industrial multimodal defect detection task.

[0023] This embodiment discloses a precise training and optimization method for intelligent algorithms based on multimodal fusion. It fully executes all steps to achieve automatic optimization of the multimodal fusion network structure, and controls the number of model parameters and inference latency while ensuring the accuracy of defect detection, thus adapting to the online detection needs of industrial production lines.

[0024] S1. Construct a multimodal dataset corresponding to the industrial defect detection target task and complete the baseline model initialization. This embodiment's dataset includes an labeled training subset and an unlabeled pre-training subset. The data originates from three modalities: visible light images, infrared thermal images, and vibration signals of workpieces synchronously collected from industrial production lines. The labeled training subset includes defect category labels and defect location annotations, while the unlabeled pre-training subset retains only the original collected data without manual annotation. Independent feature extraction networks are built for each of the three modalities: ResNet-18 is used as the image feature extraction network for visible light images and infrared thermal images, while 1D-CNN is used as the temporal feature extraction network for one-dimensional vibration time-series signals. Feature transformations are performed on the input data to output high-dimensional feature representations corresponding to each modality.

[0025] A subset of unlabeled pre-trained samples, representing 10% of the total training set, was selected. The InfoNCE cross-modal contrastive learning loss was used to constrain the low-level parameters of the three modal feature extraction networks. During the InfoNCE loss process, visible light, infrared, and vibration modal features corresponding to the same workpiece sample were treated as positive sample pairs, while any modal combination between different workpiece samples was treated as a negative sample pair. In the high-dimensional semantic space, the feature vectors of positive sample pairs were brought closer together, while the feature vectors of negative sample pairs were pushed further apart, thus achieving alignment of feature spaces across different modalities. After 20 rounds of pre-training, all trainable low-level parameters of the first 80% of the network layers closest to the input side of each modal feature extraction network were frozen, while only the top-level parameters of the last 20% of the fully connected layers closest to the output side were made available for subsequent supervised fine-tuning.

[0026] Single-modal supervised fine-tuning was performed using labeled training subset samples. After each training round, the defect recognition accuracy of visible light single-modality, infrared single-modality, and vibration signal single-modality was verified separately. A pass / fail criterion was set: the decrease in single-modal accuracy after fine-tuning relative to the single-modal accuracy after pre-training should not exceed 1%. The logic behind this threshold setting is to avoid destroying the basic modal features learned in the pre-training stage during the fine-tuning process and to prevent excessive degradation of the basic single-modal recognition ability. If the decrease in accuracy of any single-modality exceeds this threshold, the learning rate in the supervised learning stage is reduced, and fine-tuning is repeated until the criterion is met, completing the baseline model initialization.

[0027] S2. Complete the standardized mapping of the optimization space. The number of multimodal fusion layers, the feature dimension of each layer, the modal interaction method, and the feature transfer path are encoded into a discretized action space. The feature transfer path specifically refers to the configuration parameters of cross-layer residual connections between fusion layers, including three types of discrete options: whether the residual connection is enabled, the source layer number of the residual connection, and the target layer number. This embodiment, for industrial detection tasks, additionally adds a temporal window length dimension to the action space for configuring sliding window sampling of vibration temporal signals.

[0028] The validation set defect detection accuracy, overall model parameter count, and single-sample inference latency are used as model evaluation metrics, collectively forming a continuous state space. A discrete-continuous dual-domain embedding coding method is employed to complete the optimization space standardization mapping, according to the formula... Calculate the continuous action embedding vector. Embed the output continuous action vector; is a discrete action vector, which is the vector obtained after encoding each discrete parameter of the fusion structure; The matrix is ​​a learnable embedding matrix, and its dimensions are pre-defined based on the dimensions of the discrete action vectors and are updated synchronously with network training. The prior constraint weight is set to 0.2 in this embodiment. This weight is used to control the degree of influence of the modal statistical prior vector on the continuous embedding results. If the weight is too large, it will over-constrain the exploration range of the agent, and if the weight is too small, the inherent statistical information of the modality will be lost. This is a modal statistical prior vector, composed of the statistical values ​​of the feature mean and feature variance of each of the three modalities. This vector is statistically calculated after inference of all validation set samples from the baseline model and does not participate in gradient updates. This encoding method maps discrete fusion structure parameters to a continuous vector space, facilitating policy computation by the reinforcement learning agent.

[0029] S3. Constructing a proximal policy optimization agent and an adaptive weighted reward function. A reinforcement learning agent is constructed using the Proximal Policy Optimization (PPO) algorithm. The agent's input is the continuous state space information obtained from S2, and its output is the fused structured action corresponding to the discrete action space. A weighted combined reward function is constructed, and the total reward is calculated using the following formula: . Represents the total reward value of the current iteration step; To validate the normalized values ​​of performance metrics for the industrial defect detection task, performance metrics include defect detection accuracy and small target recall, where small target recall is... The internal weight is set to be no less than 30%, and the original indicators are first mapped to the 0-1 range to complete the normalization; This is the normalized value of the model parameters; the larger the number of parameters, the higher this value. This is the normalized value of single-sample inference delay; the longer the inference time, the higher this value. , , The first , , The dynamic weight coefficients corresponding to each iteration. Dynamic weight adjustment logic: early iteration phase. Prioritize higher values ​​to ensure detection performance, and gradually improve them in the later stages of iteration. and Weighting is applied to increase the penalties for constraints on parameter quantity and inference latency, thereby driving the agent to search for a multimodal fusion strategy that simultaneously satisfies detection accuracy, parameter quantity limits, and production line inference latency limits. Production line constraints pre-set upper limits for parameter quantity and single-sample inference latency. When the parameter quantity or latency exceeds the preset upper limit, the penalty term will significantly reduce the total reward, guiding the agent to avoid fusion structures that do not meet engineering implementation constraints.

[0030] S4. Initialize the evolutionary population, and iteratively optimize using reinforcement learning and evolutionary algorithms alternately. An evolutionary population containing several fusion strategy individuals is initialized, with each strategy individual corresponding to a complete set of multimodal fusion structure action sequences. All individuals in the initial population are instantiated into a network under the baseline model framework, and the initial high-quality strategy set is obtained through selection, crossover, and mutation operations and stored in the pre-training experience pool of the reinforcement learning agent.

[0031] Fitness of individuals within an evolutionary population is calculated using a modal complementarity weighted method, with the following formula: F represents the individual fitness of the fusion strategy; P represents the original performance index of the instantiated model of this strategy on the validation set for defect detection. The complementarity influence coefficient is set to 0.3 in this embodiment to adjust the contribution ratio of modal complementarity to fitness. This is the modal complementarity coefficient, which is obtained by calculating the mutual information between output features of different modalities: the lower the mutual information between two types of modal features, the stronger the modal complementarity. The higher the value, the better the overall performance of the fusion strategy. Mutual information calculation is based on the high-dimensional features of each modality output by the baseline model on the validation set, eliminating the need to retrain the network.

[0032] According to the preset iteration trigger conditions: an evolutionary population update operation is performed every 15 rounds of reinforcement learning iterations. In the selection phase, individuals are sorted by fitness from high to low, and the top 20% of individuals are selected as high-quality parents. In the crossover phase, single-point crossover is used, randomly selecting an action dimension as the crossover site, swapping all parameters after the crossover site between two parent individuals to generate offspring individuals, with the number of offspring equal to the number of high-quality parents. In the mutation phase, Gaussian mutation is performed on all offspring individuals, randomly selecting three action dimensions as mutation sites, and applying Gaussian noise perturbation to the discrete parameters corresponding to the sites. Afterwards, high-quality parent individuals and mutated offspring individuals are mixed, and individuals with lower fitness are removed to ensure that the population size after the update remains consistent with the initial population. High-quality policies in the new population are simultaneously added to the reinforcement learning experience pool to expand the policy search space.

[0033] S5. Decode the optimal fusion strategy and perform end-to-end training with a stepped learning rate. When the optimization iteration reaches the preset maximum number of iterations, or the validation set performance reaches the preset performance threshold for multiple consecutive rounds, the optimization process terminates. The action sequence with the highest total reward is selected, and a multimodal fusion network is obtained through a hierarchical topology mapping mechanism. During decoding, the mapping proceeds hierarchically as follows: First, the number of fusion layers within the action sequence is mapped to the fusion network depth; second, the feature dimension parameters of each layer are mapped to the number of output channels of the network layer; third, the modal interaction method parameters are mapped to the corresponding fusion operators, the concatenation parameters to the channel dimension concatenation operators, the element-wise addition parameters to the element-wise summation operators, and the attention parameters to the single-head cross-attention operators; fourth, the feature propagation path parameters are mapped to the residual connection branches between layers. The decoding process verifies the feature dimensions of adjacent layers in real time. If a dimension mismatch occurs, a 1×1 convolutional layer is automatically inserted to complete the dimension adaptation, outputting a directly executable neural network computation graph. The fusion network is then concatenated with the output layers of the visible light, infrared, and vibration modal feature extraction networks.

[0034] End-to-end training was conducted, with a total of 40 training epochs, divided into two phases: the first 10 epochs froze the underlying parameters of the feature extraction network, updating only all parameters of the fusion network; the remaining 30 epochs unfroze all network parameters for global fine-tuning. A validation set evaluation was performed every 8 epochs. If the performance improvement was less than 0.2% for two consecutive epochs, learning rate decay was triggered, with a maximum of 3 decays. After reaching the upper limit, the learning rate remained unchanged. The logic behind this 0.2% threshold is as follows: considering the noise in industrial detection task data, a small improvement indicates that the model is close to convergence, and there is no need to maintain a high learning rate. A gradient clipping threshold of 1.0 was set during training to avoid gradient explosion; the L2 weight decay coefficient was set to 1×10⁻⁻⁶. 4 To suppress overfitting, if the validation set performance declines for three consecutive rounds, the model is deemed overfitted, and training is terminated prematurely. For industrial multimodal detection tasks, the spatial resolution of images and infrared two-dimensional modalities is unified through affine transformation and bilinear interpolation; the vibration one-dimensional time-series signal is time-stamped and the alignment accuracy is controlled within 1ms; during the training phase, small defect samples are oversampled 3:1 to increase the learning weight of small defects.

[0035] S6, cross-scenario incremental update When the production line changes the workpiece model, the workpiece defect distribution and modal data distribution shift, triggering an incremental model update. A two-stage migration calibration mechanism is implemented: In the first stage, the parameters of the bottom-level policy network of the reinforcement learning agent are frozen, with only the top-level decision head parameters fine-tuned. The model is iterated for 15 rounds using the validation set of the new workpiece to adapt to the new scenario's modal distribution and performance evaluation standards. In the second stage, the first 20% of the parameters of the bottom-level policy network are unfrozen, and 10 rounds of fine-tuning are performed based on the experience gained from exploring new scenario policies. The historical policy experience pool is updated, retaining 70% of the original high-quality policy samples and adding 30% of the new scenario policy samples for mixed storage. A trigger condition is set: if the number of labeled samples in the new scenario exceeds 60% of the number of labeled samples in the original scenario, indicating sufficient new scenario data, a complete full-scale optimization process is initiated to re-search for the optimal fusion strategy.

[0036] In summary, this embodiment addresses the industrial multimodal defect detection scenario by constructing a multimodal dataset encompassing images, infrared signals, and vibration signals. Baseline model initialization is achieved through cross-modal comparative learning. A standardized optimization space with discrete-continuous dual-domain embedding is built, and PPO reinforcement learning and evolutionary algorithms are iteratively combined. Relying on a modal complementarity-weighted fitness function and an adaptive dynamic weight reward function, a multimodal fusion network structure that balances detection accuracy, parameter quantity, and inference latency constraints is automatically searched. Hierarchical topology mapping is used to decode action sequences into the actual network, and a stepped learning rate scheduling is employed for end-to-end training. A two-stage transfer calibration mechanism is configured to achieve incremental updates across scenarios after workpiece switching. This solution solves the technical problems of traditional fixed fusion structures being difficult to adapt to heterogeneous multimodal data in industrial detection, having poor small defect recognition performance, and struggling to balance accuracy with production line inference overhead. The entire process parameters and thresholds are given clearly defined setting logic. Those skilled in the art can fully reproduce the technical solution based on the content described in this embodiment to complete the automatic training and optimization of the industrial workpiece multimodal defect detection model.

[0037] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A precise training and optimization method for intelligent algorithms based on multimodal fusion, characterized in that, The method comprises the following components: Step 1: Construct a multimodal dataset corresponding to the target task. The dataset includes a labeled training subset and an unlabeled pre-training subset. Build an independent feature extraction network for each modality, extract high-dimensional feature representations of the data from each modality, and complete the initialization of the baseline model. Step 2: Encode the number of multimodal fusion layers, feature dimensions of each layer, modal interaction methods, and feature propagation paths into a discretized action space. Encode the validation set performance, number of parameters, and inference latency corresponding to the current fusion structure into a continuous state space. Complete the standardized mapping of the optimization space. The feature propagation path is the configuration parameter of the cross-layer residual connection between fusion layers. Step 3: Construct a reinforcement learning optimization agent using a proximal policy optimization algorithm. Construct a weighted combined reward function that includes core task performance terms and model constraint penalty terms to drive the agent to explore a fusion strategy that meets both performance and constraint requirements. Step 4: Initialize an evolutionary population containing multiple fusion strategy individuals. Generate an initial set of high-quality strategies as a pre-training experience pool for the agent through selection, crossover, and mutation operations. After every 15 rounds of reinforcement learning iterations, perform an evolutionary population update operation to expand the search space. Step 5: When the optimization process reaches the preset iteration rounds or performance threshold, select the current optimal action sequence and decode it into the corresponding multimodal fusion network structure. Then, concatenate the fusion network with the feature extraction networks of each modality and use a step-by-step learning rate decay strategy to complete end-to-end training. Step 6: When the target task or data distribution changes, the agent is fine-tuned based on the trained reinforcement learning agent network parameters and historical policy experience pool to generate a fusion policy adapted to the new scenario, thus completing the incremental update and cross-scenario transfer of the model.

2. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The baseline model initialization in step one is implemented as follows: Select samples from an unlabeled pre-training subset in the training set, accounting for 10% of the total training set samples. Use InfoNCE cross-modal contrastive learning loss to constrain the low-level parameters of each modality feature extraction network. Bring the different modality features corresponding to the same multimodal sample closer in the semantic space and push the modality features of different samples further apart. After 20 rounds of pre-training, freeze the low-level parameters of each modality feature extraction network. The low-level parameters refer to the trainable parameters of the first 80% of the network layers closest to the input side. Only the top-level parameters are retained for subsequent supervised fine-tuning. The top-level parameters refer to the parameters of the last 20% of the fully connected layers closest to the output side. The fine-tuning process uses samples from the labeled training subset. After each round of training, verify the accuracy of the single-modality task. When the decrease in accuracy of the single-modality task after pre-training does not exceed 1%, the baseline model initialization is deemed qualified.

3. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The optimization space normalization mapping in step two adopts a discrete-continuous dual-domain embedding coding method, and the continuous action embedding vector is calculated using the following formula: ,in Embed the output continuous action vector; It is a discrete action vector; For learnable embedding matrices; These are the prior constraint weights; This is the modal statistical prior vector.

4. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The weighted combination reward function in step three employs an adaptive dynamic weighting mechanism, and the total reward value is calculated using the following formula: ,in This represents the total reward value for the current iteration step. This represents the normalized performance metric of the target task on the validation set. These are the normalized values ​​of the model parameters; This is the normalized value of the model's single-sample inference latency; , , For the first The dynamic weight coefficients corresponding to each iteration.

5. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The evolutionary population selection operation in step four employs a fitness calculation method weighted by modality complementarity, using the following formula to calculate the individual fitness value: ,in The fitness value of an individual using the fusion strategy; These are the original values ​​of the performance metrics of the model corresponding to this strategy on the validation set; The complementarity influence coefficient; This is the modal complementarity coefficient.

6. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The evolutionary crossover and mutation operations in step four are performed according to the following rules: After every 15 rounds of reinforcement learning iterations, an update operation is performed on the current population. In the selection phase, the fitness value is used as the sorting criterion, and the top 20% of individuals in the population are selected as high-quality parent individuals. In the crossover phase, a single-point crossover method is used, and an action dimension is randomly selected as the crossover site. All parameters after the crossover site of two high-quality parent individuals are swapped, and the number of offspring individuals generated is the same as the number of high-quality parent individuals. In the mutation phase, Gaussian mutation is applied to the newly generated offspring individuals, and three action dimensions are randomly selected as mutation sites. Finally, the high-quality parent individuals and the mutated offspring individuals are mixed to form a new population. The population size is consistent with the initial population size, and the high-quality strategies in the new population are simultaneously stored in the reinforcement learning experience pool.

7. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The optimal fusion strategy decoding in step five adopts a hierarchical topology mapping mechanism, performing mapping operations sequentially according to the hierarchical order. First, the number of fusion layers in the action sequence is mapped to the depth structure of the fusion network, with a one-to-one correspondence between the number of layers and the parameter values. Then, the feature dimension parameters of each layer are mapped to the number of output channels of the corresponding network layer. Next, the modal interaction mode parameters are mapped to the corresponding fusion operators. The concatenation parameters correspond to the channel dimension concatenation operator, the element-wise addition parameters correspond to the element-wise summation operator, and the attention parameters correspond to the single-head cross-attention operator. Finally, the feature transfer path parameters are mapped to the residual connection branches between layers. During the mapping process, the feature dimension matching of adjacent layers is automatically checked. When a dimension mismatch occurs, a 1×1 convolutional layer is inserted for dimension adaptation. Finally, a neural network computation graph that can be directly called is generated and directly connected to the output layer of each modality feature extraction network.

8. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The end-to-end training in step five employs a performance-driven, stepped learning rate dynamic scheduling strategy. The total number of training rounds is set to 40. The training process is divided into two phases: the first 10 rounds freeze the underlying parameters of each modality feature extraction network, training only the parameters of the fusion network; the next 30 rounds unfreeze all network parameters for global fine-tuning. A validation set performance evaluation is performed every 8 rounds. If the performance improvement is less than 0.2% for two consecutive rounds, learning rate decay is triggered, with a cumulative decay count not exceeding 3 times. After reaching the decay cap, the current learning rate is maintained until training ends. During training, the gradient clipping threshold is set to 1.0, and the L2 weight decay coefficient is set to 1 × 10⁻⁻⁻⁶. 4 If the performance of the validation set declines for three consecutive rounds, the training process will be terminated early.

9. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, The incremental update and cross-scenario migration in step six adopts a two-stage migration calibration mechanism: In the first stage, the underlying policy network parameters of the reinforcement learning agent are frozen, and only the top-level decision head parameters are fine-tuned. Based on the validation set data of the new scenario, 15 rounds of iteration are performed to adapt to the performance evaluation standards and modal feature distribution of the new scenario. In the second stage, the top 20% of the parameters in the underlying policy network are unfrozen, and 10 rounds of fine-tuning are performed in combination with the policy exploration experience of the new scenario. During the iteration process, the historical policy experience pool is updated, retaining 70% of the historical high-quality policy samples in the original experience pool and supplementing 30% of the new scenario policy samples. The mixture is used as the experience basis for subsequent iterations. When the number of labeled samples in the new scenario exceeds 60% of the number of labeled samples in the original scenario, the full optimization process is started to search for the optimal fusion strategy again.

10. The method for precise training and optimization of intelligent algorithms based on multimodal fusion according to claim 1, characterized in that, For different task scenarios, corresponding adaptation processing is adopted. For temporal modal tasks that include voice and video, a temporal window length dimension is added to the action space. During the training phase, a sliding window sampling method is used to generate training samples for temporal modal data. Sine and cosine position encoding is added to the input layer of the corresponding temporal modality in the fusion network, and the encoding dimension is consistent with the feature dimension of the corresponding modality. For industrial multimodal detection tasks, affine transformation and bilinear interpolation are used to unify the spatial resolution for two-dimensional modalities such as images and infrared. For one-dimensional modalities such as vibration and soundprint, timestamp synchronization processing is used, and the time alignment accuracy is controlled within 1ms. During the training phase, small target samples are oversampled at a ratio of 3:1, and the recall rate of small targets accounts for no less than 30% of the weight of the performance reward item.