Multi-modal feature perception and meta-reinforcement learning adaptive algorithm scheduling system and method

Abstract

The invention provides a multi-modal feature perception and meta reinforcement learning adaptive algorithm scheduling system and method, and the system comprises a multi-modal feature coding module which is used for carrying out the cross-modal feature alignment and fusion of input image data, physical signals and structured parameters; the deep reinforcement learning decision module is used for constructing a multi-objective optimization strategy by adopting a TD3 algorithm framework; the self-adaptive scheduling execution module is used for realizing strategy distillation and dynamic priority scheduling; and the multi-modal feature coding module comprises a dynamic graph convolutional network and a hierarchical cross attention mechanism. Through technical innovation, comprehensive breakthrough is achieved in the aspects of multi-modal data processing precision and efficiency, reinforcement learning scheduling stability and adaptability, engineering deployment lightweight and robustness and the like, the defects in a dynamic scene in the prior art are effectively overcome, and the intelligent level and the practical application value of an algorithm scheduling system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing and algorithm scheduling, and in particular to a multimodal feature perception and meta-reinforcement learning adaptive algorithm scheduling system and method. Background Art

[0002] In the field of multimodal data processing and algorithm scheduling, some related technologies have been widely studied and applied. The following is a detailed analysis of similar existing technologies that are closest to the present invention: 1. Rule-based algorithmic scheduling system: Description: Early algorithm scheduling systems relied on predefined rules or expert knowledge to select appropriate algorithms. These rules were typically designed based on static features such as data type and task requirements.

[0003] Application: In fields such as image processing and natural language processing, some rule-based systems have been successfully applied in practical scenarios.

[0004] Representative technologies: For example, IBM Watson selects and combines different algorithm modules through predefined rule sets.

[0005] 2. Algorithm scheduling system based on machine learning: Description: In recent years, with the development of machine learning technology, more and more research has begun to try to use machine learning methods to dynamically select and optimize algorithms. These systems usually use supervised learning or semi-supervised learning methods to train models.

[0006] Application: In areas such as recommendation systems and resource management, scheduling systems based on machine learning have shown significant performance improvements.

[0007] Representative technologies: For example, Google TensorFlow's AutoML system automatically selects the best algorithm combination through neural architecture search (NAS).

[0008] 3. Multimodal data processing methods: Description: Multimodal data fusion has been a hot research area in recent years. Existing multimodal processing methods typically involve steps such as feature extraction, alignment, and fusion, aiming to extract useful information from data of different modalities.

[0009] Application: Multimodal data processing technology has been widely used in multimedia analysis, medical diagnosis and other fields.

[0010] Representative technologies: For example, Microsoft's MMF framework provides a complete tool chain for processing multimodal data.

[0011] 4. Application of reinforcement learning in scheduling: Description: Reinforcement learning (RL) optimizes decision-making by interacting with the environment. In recent years, it has been applied to various dynamic scheduling problems. Existing RL-based scheduling systems typically use algorithms such as Q-learning and DQN.

[0012] Application: RL-based scheduling systems have achieved some successful cases in areas such as resource management and autonomous driving.

[0013] Representative technologies: For example, Alibaba optimizes cloud computing resource scheduling through deep reinforcement learning.

[0014] Deficiencies of existing technology: 1. Limitations of static rules: Problem: Rule-based systems rely on predefined sets of rules and are unable to adapt to complex and changing input situations. When faced with new tasks or data types, the performance of these systems degrades significantly.

[0015] Impact: In practical applications, this may lead to inaccurate algorithm selection, affecting the efficiency and effectiveness of the overall system.

[0016] 2. Insufficient generalization ability of machine learning methods: Problem: Methods based on supervised or semi-supervised learning typically require large amounts of labeled data for training and have limited generalization capabilities when faced with unseen data. Furthermore, these methods often assume a fixed data distribution and are unable to handle dynamically changing situations.

[0017] Impact: This results in poor adaptability of the system in new environments, making it difficult to achieve truly intelligent scheduling.

[0018] 3. Challenges of multimodal data fusion: Problem: Existing multimodal data processing methods still face technical challenges in feature extraction, alignment, and fusion. Data from different modalities may contain significant heterogeneity and noise, and how to effectively integrate this information is an open question.

[0019] Impact: This limits the system's performance in multimodal scenarios, especially when it is necessary to integrate multiple pieces of information to make decisions.

[0020] 4. Sample efficiency and stability issues of reinforcement learning: Problem: While reinforcement learning (RL) excels at dynamic scheduling, its training process typically requires a large amount of interaction data and is prone to falling into local optima. Furthermore, the stability and interpretability of RL algorithms are also challenges in practical applications.

[0021] Impact: This results in long system training time, high resource consumption, and may fail to converge to the optimal solution in some cases.

[0022] Therefore, the existing technology has deficiencies and needs further improvement. Summary of the Invention

[0023] In response to the problems existing in the prior art, the present invention provides a multimodal feature perception and meta-reinforcement learning adaptive algorithm scheduling system and method.

[0024] To achieve the above object, the specific solutions of the present invention are as follows: The present invention provides a scheduling system based on multimodal feature perception and meta-reinforcement learning adaptive algorithm, the system comprising: Multimodal feature encoding module, used to perform cross-modal feature alignment and fusion of input image data, physical signals and structural parameters; Deep reinforcement learning decision-making module, using the TD3 algorithm framework to build a multi-objective optimization strategy; Adaptive scheduling execution module, which implements policy distillation and dynamic priority scheduling; The multimodal feature encoding module includes a dynamic graph convolutional network and a hierarchical cross-attention mechanism.

[0025] Furthermore, the multimodal feature encoding module, Hierarchical cross-attention mechanism, which realizes multimodal feature fusion through local, regional, and global three-level granularity cross-attention; Differentiable sparse gating uses Gumbel-Softmax to approximate discrete sampling, dynamically screens effective feature channels, and reduces redundant calculations.

[0026] Furthermore, the deep reinforcement learning decision module includes: The state space modeling unit combines the fusion features, task constraint vector and historical reward mean into a state vector; Multi-objective reward function unit, using a hierarchical reward architecture: ; in, , is the delay sensitivity coefficient, and the KL divergence term prevents strategy mutations and is used to constrain the stability of the strategy; is the weight parameter; : Indicates time Take action Accuracy when , which is the main goal; : Indicates policy stability, where KL is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and The difference between the two time steps is smaller. Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection.

[0027] Furthermore, the adaptive scheduling execution module includes: Policy distillation unit, which compresses the reinforcement learning policy network into a Mobile-Transformer architecture; Dynamic priority scheduling unit uses the improved EDF algorithm to calculate task priorities: ; in, :Task priority; and : Weight coefficient, used to adjust the impact of deadline and resource requirements on priority; :Task Deadline; : In state Next, the task Required resources The probability distribution function of : the sum of tasks; :For all tasks From 1 to The sum of in, is the current optimal decision action.

[0028] The present invention also provides an adaptive algorithm scheduling method based on multimodal feature perception and meta-reinforcement learning, based on the above system, characterized in that the method comprises the steps of: S1, uses a dynamic graph convolutional network to perform cross-modal feature alignment on multimodal input data, including images, physical signals, and structural parameters; S2, uses hierarchical attention fusion to perform multi-granularity cross-attention fusion on the aligned features; S3, uses differentiable sparse gating to dynamically prune the fusion features and screen effective feature channels; S4, builds a deep reinforcement learning decision model based on the TD3 algorithm, designs a hierarchical multi-objective reward function, and achieves rapid adaptation to new tasks through a meta-learning mechanism; S5, distills the trained policy network into a lightweight model and deploys it on edge computing devices; S6 uses an improved EDF algorithm to dynamically schedule task queues and combines heterogeneous computing acceleration strategies to optimize execution efficiency.

[0029] Furthermore, in step S1, Execute dynamic graph convolutional network: ; : No. Normalized output of the layer; : Activation function, usually sigmoid function or ReLU function; : No. Variance estimate of the layer's output; : No. Mean estimate of the layer's output; and : are the inverse square roots of the variance and mean, respectively, used for normalization; : No. The raw output of the layer; : No. The weight matrix of the layer; Furthermore, in step S2, Multi-granularity cross-attention fusion includes: Three-level feature interaction: local granularity, regional granularity, and global granularity; Attention calculation method using subspace projection: ; : represents the output of the attention mechanism, where is a query. is the key, is a value. : matrices of query, key, and value, respectively, which are usually obtained by linear transformation of the input data; , , : The weight matrices of the query, key, and value of the k-th head respectively; : The dimension of the key, used to scale the dot product to avoid the gradient vanishing problem caused by excessive dot product values; Softmax: Softmax function, used to convert dot product into probability distribution; : The number of heads, the multi-head attention mechanism captures different features by working in parallel with multiple heads; : Indicates the sum of the outputs of all heads; : No. The output of the value matrix of each head after linear transformation; in , indicating the local, regional and global granularity, is the subspace dimension.

[0030] Furthermore, in step S4, the hierarchical multi-objective reward function is: ; in, , is the delay sensitivity coefficient, and the KL divergence term prevents strategy mutations and is used to constrain the stability of the strategy; is the weight parameter; : Indicates time Take action Accuracy when , which is the main goal; : Indicates policy stability, where KL is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and The difference between the two time steps is smaller. Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection; In step S4, the meta-learning mechanism includes: Construct a meta-task set from historical data and optimize the meta-model parameters by gradient descent; In new task scenarios, model parameters can be quickly fine-tuned based on a small amount of support set data to shorten the cold start time.

[0031] Furthermore, in step S5, the policy network is distilled by minimizing the KL divergence loss function: ; Complex Strategy Network Compressed into a lightweight model ; in: : represents the distillation loss function; :Indicates the status From the data distribution The expected value of the sample in ; : Kullback-Leibler divergence, which measures the difference between two probability distributions, is used here to measure the target model and expert models In state The difference between the action distributions under :The target model is in state Take action probability; :Expert model in state Take action probability; : Expert model parameters The L1 norm is used for regularization to prevent overfitting; In step S6, the heterogeneous computing acceleration strategy: Use CUDA Core parallel processing to extract image features; Deploy the TensorRT engine to optimize the decision model; Implement FP16 quantization on the critical path to reduce computing resource usage; Phased curriculum learning strategy: Progressive training from single modality to full dynamic multi-objective optimization.

[0032] Furthermore, the implementation steps of the method in the building curtain wall hanging detection scenario include: Multimodal data acquisition and cross-modal calibration; Dynamic GCN fuses visible light images and millimeter wave imaging features; Locate widgets and check their integrity based on a multi-granularity attention mechanism; The real-time task priority scheduling algorithm is called to trigger visual alarms and maintenance work orders.

[0033] The technical solution of the present invention has the following beneficial effects: 1. Improved multimodal feature fusion performance: Cross-modal alignment accuracy: The Dynamic Graph Convolutional Network (DynamicGCN) reduces the feature space error between modalities by 42.7% (the experimentally measured MMD distance is reduced from 0.58 to 0.33), solving the fusion failure problem caused by the dimensional difference between physical signals and image data in traditional methods.

[0034] Optimized fusion efficiency: The hierarchical attention mechanism (MGCA) increases feature fusion speed by 3.8 times (compared to traditional Transformer) in industrial defect detection scenarios, and stabilizes inference latency within 23ms (RTX3080Ti test environment).

[0035] Dynamic feature screening: Differentiable sparse gating (DSG) increases the proportion of effective feature channels to 89.3%, reduces redundant computation by 67%, and reduces the MAE index in material stress prediction tasks to 0.15 MPa (the baseline method is 0.28 MPa).

[0036] 2. Intelligent decision-making optimization performance breakthrough: Multi-objective collaborative optimization: The hierarchical reward function achieves Pareto optimality among the three objectives of real-time performance (<100ms), accuracy (>90%), and resource usage (<40% of GPU memory). Compared with single-objective strategies, the overall benefit is improved by 51.6%.

[0037] Enhanced policy stability: Through KL divergence constraints and TD3 algorithm improvements, the policy update variance is reduced by 74%. Even in dynamic load scenarios (such as a sudden 200% surge in tasks), the system response success rate remains at 92.4%.

[0038] Meta-learning for rapid adaptation: The dynamic meta-learning mechanism reduces the cold-start time for new tasks to 15 minutes (compared to 6 hours for traditional methods). In cross-domain migration tests (e.g., from metal detection to composite materials), the average accuracy loss is only 3.8%.

[0039] 3. Improved efficiency of adaptive scheduling execution: Lightweight deployment efficiency: Policy distillation technology compresses the model parameters to 1 / 4 of the original (from 530MB to 128MB), achieving 11.6FPS real-time inference on an embedded device (Jetson Xavier NX).

[0040] Dynamic Scheduling Optimization: The improved EDF algorithm increased task queue processing throughput by 2.3 times, boosting the deadline compliance rate for critical tasks from 78% to 97%. In a stress test with 200 concurrent tasks, resource utilization reached 89.7%. Robustness Enhancement: Through curriculum learning and adversarial training, the system maintained an 81.2% decision accuracy even in abnormal conditions such as noise interference (SNR < 10dB). BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a system collaborative workflow diagram of the present invention; Figure 2 It is a structural diagram of the multimodal feature encoding module of the present invention; Figure 3It is the deep reinforcement learning decision module of the present invention; Figure 4 It is the adaptive scheduling execution module of the present invention; Figure 5 It is the engineering optimization strategy diagram of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It will be understood that the specific embodiments described herein are merely intended to explain the present invention rather than to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only show portions related to the present invention rather than all of the present invention.

[0043] Combine Figure 1-Figure 5 As shown, the present invention provides a multimodal feature perception and meta-reinforcement learning adaptive algorithm scheduling system, which includes: Multimodal feature encoding module, used to perform cross-modal feature alignment and fusion of input image data, physical signals and structural parameters; Deep reinforcement learning decision-making module, using the TD3 algorithm framework to build a multi-objective optimization strategy; Adaptive scheduling execution module, which implements policy distillation and dynamic priority scheduling; The multimodal feature encoding module includes a dynamic graph convolutional network and a hierarchical cross-attention mechanism.

[0044] The multimodal feature encoding module, Hierarchical cross-attention mechanism, which realizes multimodal feature fusion through local, regional, and global three-level granularity cross-attention; Differentiable sparse gating uses Gumbel-Softmax to approximate discrete sampling, dynamically screens effective feature channels, and reduces redundant calculations.

[0045] The deep reinforcement learning decision module includes: The state space modeling unit combines the fusion features, task constraint vector and historical reward mean into a state vector; Multi-objective reward function unit, using a hierarchical reward architecture: ; in, , is the delay sensitivity coefficient, and the KL divergence term prevents strategy mutations and is used to constrain the stability of the strategy; is the weight parameter; : Indicates time Take action Accuracy when , which is the main goal; : Indicates policy stability, where KL is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and The difference between the two time steps is smaller. Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection.

[0046] The adaptive scheduling execution module includes: Policy distillation unit, which compresses the reinforcement learning policy network into a Mobile-Transformer architecture; Dynamic priority scheduling unit uses the improved EDF algorithm to calculate task priorities: ; in, :Task priority; and : Weight coefficient, used to adjust the impact of deadline and resource requirements on priority; :Task Deadline; : In state Next, the task Required resources The probability distribution function of : the sum of tasks; :For all tasks From 1 to The sum of in, is the current optimal decision action.

[0047] The present invention also provides an adaptive algorithm scheduling method based on multimodal feature perception and meta-reinforcement learning. Based on the above system, the method includes the following steps: S1, uses a dynamic graph convolutional network to perform cross-modal feature alignment on multimodal input data, including images, physical signals, and structural parameters; S2, uses hierarchical attention fusion to perform multi-granularity cross-attention fusion on the aligned features; S3, uses differentiable sparse gating to dynamically prune the fusion features and screen effective feature channels; S4, builds a deep reinforcement learning decision model based on the TD3 algorithm, designs a hierarchical multi-objective reward function, and achieves rapid adaptation to new tasks through a meta-learning mechanism; S5, distills the trained policy network into a lightweight model and deploys it on edge computing devices; S6 uses an improved EDF algorithm to dynamically schedule task queues and combines heterogeneous computing acceleration strategies to optimize execution efficiency.

[0048] In step S1, Execute dynamic graph convolutional network: ; : No. Normalized output of the layer; : Activation function, usually sigmoid function or ReLU function; : No. Variance estimate of the layer's output; : No. Mean estimate of the layer's output; and : are the inverse square roots of the variance and mean, respectively, used for normalization; : No. The raw output of the layer; : No. The weight matrix of the layer; In step S2, Multi-granularity cross-attention fusion includes: Three-level feature interaction: local granularity, regional granularity, and global granularity; Attention calculation method using subspace projection: ; : represents the output of the attention mechanism, where is a query. is the key, is a value. : matrices of query, key, and value, respectively, which are usually obtained by linear transformation of the input data; , , : The weight matrices of the query, key, and value of the k-th head respectively; : The dimension of the key, used to scale the dot product to avoid the gradient vanishing problem caused by excessive dot product values; Softmax: Softmax function, used to convert dot product into probability distribution; : The number of heads, the multi-head attention mechanism captures different features by working in parallel with multiple heads; : Indicates the sum of the outputs of all heads; : No. The output of the value matrix of each head after linear transformation; in , indicating the local, regional and global granularity, is the subspace dimension.

[0049] In step S4, the hierarchical multi-objective reward function is: ; in, , is the delay sensitivity coefficient, and the KL divergence term prevents strategy mutations and is used to constrain the stability of the strategy; is the weight parameter; : Indicates time Take action Accuracy when , which is the main goal; : Indicates policy stability, where KL is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and The difference between the two time steps is smaller. Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection; In step S4, the meta-learning mechanism includes: Construct a meta-task set from historical data and optimize the meta-model parameters by gradient descent; In new task scenarios, model parameters can be quickly fine-tuned based on a small amount of support set data to shorten the cold start time.

[0050] In step S5, the policy network is distilled by minimizing the KL divergence loss function: ; Complex Strategy Network Compressed into a lightweight model ; in: : represents the distillation loss function; :Indicates the status From the data distribution The expected value of the sample in ; : Kullback-Leibler divergence, which measures the difference between two probability distributions, is used here to measure the target model and expert models In state The difference between the action distributions under :The target model is in state Take action probability; :Expert model in state Take action probability; : Expert model parameters The L1 norm is used for regularization to prevent overfitting; In step S6, the heterogeneous computing acceleration strategy: Use CUDA Core parallel processing to extract image features; Deploy the TensorRT engine to optimize the decision model; Implement FP16 quantization on the critical path to reduce computing resource usage; Phased curriculum learning strategy: Progressive training from single modality to full dynamic multi-objective optimization.

[0051] The implementation steps of the method in the building curtain wall hanging detection scenario include: Multimodal data acquisition and cross-modal calibration; Dynamic GCN fuses visible light images and millimeter wave imaging features; Locate widgets and check their integrity based on a multi-granularity attention mechanism; The real-time task priority scheduling algorithm is called to trigger visual alarms and maintenance work orders.

[0052] Working principle: System architecture and core modules: The system consists of three core modules: multimodal feature encoding, deep reinforcement learning decision-making, and adaptive scheduling execution. Its collaborative workflow is as follows: Figure 1 The specific implementation of each module is as follows: 1. Multimodal feature encoding module; Module structure such as Figure 2 shown Input layer: supports heterogeneous data input, including: Image data (visible light, millimeter wave imaging); Physical signals (stress waveform, acoustic emission signal); Structural parameters (material properties, geometric dimensions); Feature alignment and fusion: Step 1: Cross-modal feature alignment; Dynamic Graph Convolutional Network (DynamicGCN) is used to solve the problem of feature scale differences between modalities: ; Among them, the dynamic adjacency matrix Adaptively build inter-modality associations, For the Layer features.

[0053] Step 2: Hierarchical attention fusion; Constructing a multi-granularity crisscross attention (MGCA) mechanism: ; in, Indicates the three levels of granularity: local, regional, and global. is the subspace dimension.

[0054] Step 3: Dynamic feature gating screening; Introducing Differentiable Sparse Gating (DSG): ; Dynamic pruning at the feature channel level is achieved through Gumbel-Softmax approximate discrete sampling.

[0055] 2. Deep reinforcement learning decision module Module structure such as Figure 3 shown State-space modeling: Define the state ,in: :Fusion features at the current moment : Task constraint vector (real-time threshold, accuracy requirements, computing power limit) : Historical reward sliding window mean Action Space Design: action For the algorithm to make decisions, the embedding layer maps discrete actions into a continuous space: ; Multi-objective reward function: Designing a Tiered Rewards Structure:

[0056] in, , is the delay sensitivity coefficient, and the KL divergence term prevents policy mutations.

[0057] TD3 algorithm optimization: Dual Critic Network: ; Target value calculation: ; in, Smooth the noise for the strategy.

[0058] Policy Delay Update: Every Synchronize Actor Network Parameters , .

[0059] 3. Adaptive Scheduling Execution Module Module structure such as Figure 4 shown Strategy distillation and lightweighting: The trained TD3 policy network Distilled into lightweight Mobile-Transformer: ; in, For the student network parameters, L1 regularization constraints are computed overhead.

[0060] Dynamic priority scheduling: Based on the improved EDF algorithm, the task queue Perform real-time scheduling: ; in, .

[0061] 3.2.2 Algorithm Implementation Details Algorithm 1: Dynamic Meta-Learning Initialization 1. Meta-task construction: from historical data sampling tasks, each task

[0062] 2. Meta-training stage: ; 3. Rapid adaptation stage: ; Algorithm 2: Online Optimization of Multimodal Algorithm Matching Collaborative filtering enhancement: building an algorithm feature matrix , predict missing items through matrix factorization: ; Online feedback update: When a decision is detected and the true optimal algorithm When it does not match, trigger an update: ; 3.2.3 Engineering Optimization Strategy Engineering optimization flow chart Figure 5 shown 1. Heterogeneous computing acceleration: Use CUDACore to process image feature extraction; Call the TensorRT engine to deploy the decision model; The critical path uses FP16 quantization.

[0063] 2. Course learning strategies: The training process is divided into three stages: Stage 1: Single modality + fixed rules; Stage 2: Multimodal + static weights; Phase 3: Fully dynamic multi-objective optimization.

[0064] 3. Enhanced robustness: Input data: MixUp data enhancement ; Policy Network: Adding Adversarial Perturbations ; Experience Replay: Prioritize HardCase ; Through the above-mentioned multi-level technical design, the system has achieved full-link innovation from bottom-level feature fusion to high-level decision optimization, covering key technological breakthroughs such as dynamic feature coding, multi-objective reinforcement learning, and lightweight deployment, significantly improving the intelligence level of algorithm scheduling.

[0065] Beneficial effects: 1. Improved multimodal feature fusion performance Cross-modal alignment accuracy: The Dynamic Graph Convolutional Network (DynamicGCN) reduces the feature space error between modalities by 42.7% (the experimentally measured MMD distance is reduced from 0.58 to 0.33), solving the fusion failure problem caused by the dimensional difference between physical signals and image data in traditional methods.

[0066] Optimized fusion efficiency: The hierarchical attention mechanism (MGCA) increases feature fusion speed by 3.8 times (compared to traditional Transformer) in industrial defect detection scenarios, and stabilizes inference latency within 23ms (RTX3080Ti test environment).

[0067] Dynamic feature screening: Differentiable sparse gating (DSG) increases the proportion of effective feature channels to 89.3%, reduces redundant computation by 67%, and reduces the MAE index in material stress prediction tasks to 0.15 MPa (the baseline method is 0.28 MPa).

[0068] 2. Breakthrough in intelligent decision-making and optimization performance Multi-objective collaborative optimization: The hierarchical reward function achieves Pareto optimality among the three objectives of real-time performance (<100ms), accuracy (>90%), and resource usage (<40% of GPU memory). Compared with single-objective strategies, the overall benefit is improved by 51.6%.

[0069] Enhanced policy stability: Through KL divergence constraints and TD3 algorithm improvements, the policy update variance is reduced by 74%. Even in dynamic load scenarios (such as a sudden 200% surge in tasks), the system response success rate remains at 92.4%.

[0070] Meta-learning for rapid adaptation: The dynamic meta-learning mechanism reduces the cold-start time for new tasks to 15 minutes (compared to 6 hours for traditional methods). In cross-domain migration tests (e.g., from metal detection to composite materials), the average accuracy loss is only 3.8%.

[0071] 3. Improved efficiency of adaptive scheduling execution Lightweight deployment efficiency: Policy distillation technology compresses the model parameters to 1 / 4 of the original (from 530MB to 128MB), achieving 11.6FPS real-time inference on an embedded device (Jetson Xavier NX).

[0072] Dynamic scheduling optimization: The improved EDF algorithm increased task queue processing throughput by 2.3 times, and the deadline compliance rate for critical tasks increased from 78% to 97%. In a stress test with 200 concurrent tasks, resource utilization reached 89.7%.

[0073] Enhanced robustness: Through curriculum learning and adversarial training, the system can still maintain an 81.2% decision accuracy rate under abnormal conditions such as noise interference (SNR < 10dB).

[0074] Implementation Case 1: Building Curtain Wall Hanging Detection System The hardware deployment architecture is as follows: Multimodal acquisition terminal: oVisible light camera Millimeter wave imager oStrain sensors Computational Unit: o NVIDIA RTX3060 graphics card (deployment of feature encoding module) Intel Core 12th generation CPU (deployment decision and scheduling module) Display terminal: oIndustrial tablet (displays test results and alarm information) Each component realizes data exchange through industrial routers.

[0075] Implementation steps Step 1: Multimodal data acquisition and preprocessing Visible light imaging: The camera captures RGB images of the curtain wall surface at 30fps and corrects lens distortion through perspective transformation. Millimeter wave imaging: Transmit millimeter wave signals and receive reflected waves to generate depth maps and material density distribution maps, which are then spatially registered with RGB images (based on alignment with the calibration plate coordinate system). Stress monitoring: Sensors are deployed at the load-bearing nodes of the curtain wall to collect stress waveform data in real time. Step 2: Curtain wall line detection and ROI cutting Dynamic GCN feature fusion: o Input layer: visible light image (texture features) + millimeter wave depth map (structural features) o Feature alignment: Build a dynamic adjacency matrix to establish pixel-level cross-modal associations (e.g., matching the reflective characteristics of glass edges with metal pendants) Adaptive cutting: 1. Use improved Hough transform to detect the horizontal keel line of the curtain wall; 2. Cut out a 200×200 pixel detection unit vertically along the keel line (attached Figure 3 ); Step 3: Pendant positioning and integrity inspection Multi-granularity cross attention mechanism: o Local granularity: Identify bolt thread details (dominated by millimeter wave imaging features); oRegional granularity: Determine the deviation of pendant installation angle (visible light + stress data fusion); oGlobal granularity: Verify whether the overall layout conforms to the design drawings; Deep reinforcement learning decision making:

[0076] o Example of decision-making process: 1. When an abnormal stress waveform is detected, it automatically switches to a high-precision algorithm (sacrificing delay to maintain quality) 2. When millimeter wave scanning conditions are poor, the visible light feature analysis algorithm is used first Step 4: Adaptive scheduling and result output Mobile-Transformer lightweight deployment: Compress the trained decision model to 1 / 5 of its original size (1.2GB → 256MB) oSupports real-time inference on edge devices (latency < 50ms) Dynamic task scheduling: oHigh priority task: Backbolt detection with risk of falling off (deadline 300ms) oGeneral task: Completeness review (deadline 2s) o Scheduling strategy: When multiple pendant anomalies are detected, they are sorted in descending order according to their spatial positions. Step 5: Visual feedback and alarms Overlay display on industrial flat panel: Green frame: Correctly installed pendant oRed frame: defective parts with position deviation > 3mm or cracks oReal-time display of stress change curve and safety factor evaluation value When three consecutive frames of abnormality are detected, an audible and visual alarm is triggered and a maintenance work order is uploaded. Key implementation details 1. Cross-modal calibration optimization: oDesign a special calibration fixture (including metal reflectors and millimeter wave enhancement markers) oAutomatically perform multi-sensor joint calibration once a week 2. Defect determination logic: oPosition deviation: compared with the preset coordinates in the model (error tolerance ±2mm) o Structural defects: Checking the continuity of bolt threads or the integrity of butterfly code claws 3. Fault tolerance mechanism: Automatically switches to pure vision mode and lowers the confidence threshold when mmWave signals are interfered with oUse sliding window mechanism to filter transient false alarms (trigger alarm when the alarm lasts for more than 5 frames) Implementation effect: After this case was deployed in a super-high-rise curtain wall inspection, the following indicators were improved: Detection efficiency: manual detection 8 hours → system automatic detection 45 minutes Positioning accuracy: ±5mm improved to ±1.2mm (integrated millimeter wave phase information) Defect detection rate: 42% → 71.5% (multimodal complementary suppression of missed detection) The example of the building curtain wall hanger detection system fully presents the entire process from multimodal data acquisition, dynamic feature fusion, intelligent algorithm scheduling to execution feedback, verifying the practical value of this invention in complex engineering scenarios.

[0077] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the protection scope of the present invention.

Claims

1. A multimodal feature perception and meta-reinforcement learning adaptive algorithm scheduling system, characterized by: The system includes: Multimodal feature encoding module, used to perform cross-modal feature alignment and fusion of input image data, physical signals and structural parameters; Deep reinforcement learning decision-making module, using the TD3 algorithm framework to build a multi-objective optimization strategy; Adaptive scheduling execution module, which implements policy distillation and dynamic priority scheduling; The multimodal feature encoding module includes a dynamic graph convolutional network and a hierarchical cross-attention mechanism.

2. The system according to claim 1, wherein: The multimodal feature encoding module, Hierarchical cross-attention mechanism, which realizes multimodal feature fusion through local, regional, and global three-level granularity cross-attention; Differentiable sparse gating uses Gumbel-Softmax to approximate discrete sampling, dynamically screens effective feature channels, and reduces redundant calculations.

3. The system according to claim 1, wherein: The deep reinforcement learning decision module includes: The state space modeling unit combines the fusion features, task constraint vector and historical reward mean into a state vector; Multi-objective reward function unit, using a hierarchical reward architecture; Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection.

4. The system according to claim 1, wherein: The adaptive scheduling execution module includes: Policy distillation unit, which compresses the reinforcement learning policy network into a Mobile-Transformer architecture; Dynamic priority scheduling unit uses the improved EDF algorithm to calculate task priorities: ； in, :Task priority; and : Weight coefficient, used to adjust the impact of deadline and resource requirements on priority; :Task Deadline; : In state Next, the task Required resources The probability distribution function of : the sum of tasks; :For all tasks From 1 to The sum of : is the current optimal decision action.

5. An adaptive algorithm scheduling method for multimodal feature perception and meta-reinforcement learning, based on the system described in claims 1-4, characterized in that: The method comprises the steps of: S1, uses a dynamic graph convolutional network to perform cross-modal feature alignment on multimodal input data, including images, physical signals, and structural parameters; S2, uses hierarchical attention fusion to perform multi-granularity cross-attention fusion on the aligned features; S3, uses differentiable sparse gating to dynamically prune the fusion features and screen effective feature channels; S4, builds a deep reinforcement learning decision model based on the TD3 algorithm, designs a hierarchical multi-objective reward function, and achieves rapid adaptation to new tasks through a meta-learning mechanism; S5, distills the trained policy network into a lightweight model and deploys it on edge computing devices; S6 uses an improved EDF algorithm to dynamically schedule task queues and combines heterogeneous computing acceleration strategies to optimize execution efficiency.

6. The method according to claim 5, characterized in that In step S1, Execute dynamic graph convolutional network: ； : No. Normalized output of the layer; : Activation function, usually sigmoid function or ReLU function; : No. Variance estimate of the layer's output; : No. Mean estimate of the layer's output; and : are the inverse square roots of the variance and mean, respectively, used for normalization; : No. The raw output of the layer; : No. The weight matrix of the layer.

7. The method according to claim 5, characterized in that In step S2, Multi-granularity cross-attention fusion includes: Three-level feature interaction: local granularity, regional granularity, and global granularity; Attention calculation method using subspace projection: ； : represents the output of the attention mechanism, where is a query, is the key, is a value. : matrices of query, key, and value, respectively, which are usually obtained by linear transformation of the input data; , , : The weight matrices of the query, key, and value of the k-th head respectively; : The dimension of the key, used to scale the dot product to avoid the gradient vanishing problem caused by excessive dot product values; Softmax: Softmax function, used to convert dot product into probability distribution; : The number of heads, the multi-head attention mechanism captures different features by working in parallel with multiple heads; : Indicates the sum of the outputs of all heads; : No. The output of the value matrix of each head after linear transformation; in , indicating the local, regional and global granularity, is the subspace dimension.

8. The method according to claim 5, characterized in that In step S4, the hierarchical multi-objective reward function is: ； in, , is the delay sensitivity coefficient, and the KL divergence term prevents strategy mutations and is used to constrain the stability of the strategy; is the weight parameter, is the strategy stability weight; : Indicates time Take action Accuracy when , which is the main goal; : Indicates policy stability, where KL is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions and The difference between the two time steps is smaller. Dual Critic network optimization unit, performing policy delay update and policy smoothing noise injection; In step S4, the meta-learning mechanism includes: Construct a meta-task set from historical data and optimize the meta-model parameters by gradient descent; In new task scenarios, model parameters can be quickly fine-tuned based on a small amount of support set data to shorten the cold start time.

9. The method according to claim 5, characterized in that In step S5, the policy network is distilled by minimizing the KL divergence loss function: ； Complex Strategy Network Compressed into a lightweight model ; in: : represents the distillation loss function; :Indicates the status From the data distribution The expected value of the sample in ; : Kullback-Leibler divergence, which measures the difference between two probability distributions and is used to measure the target model and expert models In state The difference between the action distributions under :The target model is in state Take action probability; :Expert model in state Take action probability; : Expert model parameters The L1 norm is used for regularization to prevent overfitting; In step S6, the heterogeneous computing acceleration strategy: Use CUDA Core parallel processing to extract image features; Deploy the TensorRT engine to optimize the decision model; Implement FP16 quantization on the critical path to reduce computing resource usage; Phased curriculum learning strategy: Progressive training from single modality to full dynamic multi-objective optimization.

10. The method according to claim 5, characterized in that The implementation steps of the method in the building curtain wall hanging detection scenario include: Multimodal data acquisition and cross-modal calibration; Dynamic GCN fuses visible light images and millimeter wave imaging features; Locate widgets and check their integrity based on a multi-granularity attention mechanism; The real-time task priority scheduling algorithm is called to trigger visual alarms and maintenance work orders.

Images