Dual-mode scene lightweight vertical model construction method and system, and electronic device

Abstract

The application provides a bimodal scene lightweight vertical model construction method and system and electronic equipment, by acquiring bimodal data and preprocessing, then visual-text cross-modal importance scoring is carried out, a structured pruning strategy is adopted, the pruning proportion is dynamically adjusted according to the importance score, the key visual channel and the text neuron are retained, the model is lightened, after knowledge distillation fine-tuning, directional fine-tuning is implemented, multi-target loss is designed for multi-class text tasks for optimization; through hierarchical extraction and splicing features to form comprehensive representation, fully mining the value of text information, improve the utilization efficiency of text modal, use the loss function containing cross-modal contrast loss to enhance the modal alignment, improve the detection accuracy, improve the adaptability of the model in the specific scene.

Description

Technical Field

[0001] This invention relates to the field of visibility meters, and in particular to a method, system, and electronic device for constructing a lightweight vertical category model of a dual-modal scene. Background Technology

[0002] In the process of intelligent management of construction sites, multimodal fusion technology combining visual monitoring and textual specifications has been gradually applied to core scenarios such as safety compliance inspection and equipment status recognition. By integrating image and text information, it can improve the understanding of complex construction site environments to a certain extent, providing technical support for construction site safety management and equipment monitoring. However, in practical applications, existing technologies still face significant challenges, severely restricting their large-scale promotion in construction site scenarios.

[0003] Existing technologies suffer from the following shortcomings: 1. General-purpose multimodal models typically have billions of parameters, placing extremely high demands on hardware resources. However, the core deployment equipment on construction sites is mostly edge devices, such as smart safety helmets, inspection drones, and Jetson Nano edge computing platforms, with limited computing power and storage resources. When existing large-scale models are deployed on these edge devices, inference latency is significant, failing to meet the real-time monitoring needs of construction sites and resulting in a disconnect between technology and practical application scenarios. 2. Construction site text data contains three key types of information: on-site safety standards, equipment tags, and types of safety hazards to be detected. However, existing multimodal models only use a simple text feature concatenation processing method, without designing a layered processing mechanism for the characteristics of the three types of heterogeneous text. This results in text information not being effectively transformed into model decision-making basis, ultimately leading to an excessively high false alarm rate for safety violation detection, significantly reducing the accuracy of equipment identification and hazard classification. 3. Existing general-purpose multimodal models are not optimized for the specific characteristics of construction site scenarios, resulting in insufficient model generalization ability. Meanwhile, due to the large number of model parameters, high-performance computing hardware is required for deployment, which significantly increases the cost of intelligent transformation of construction sites, limits the application of the technology in small and medium-sized construction sites and remote projects, and makes it difficult to support the comprehensive construction of smart construction sites. Summary of the Invention

[0004] The main objective of this invention is to provide a method, system, and electronic device for constructing a lightweight vertical category model for a dual-modal scene, thereby solving the technical problems of visual-text modality imbalance, poor detection accuracy, poor scene adaptability, and high engineering application cost in the prior art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for constructing a lightweight vertical category model for a dual-modal scene, comprising the following steps:

[0006] S1: Acquire dual-modal data and preprocess it, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in layers to form a comprehensive representation.

[0007] S2: Perform visual-text cross-modal importance scoring, define a joint importance function that integrates gradient information and cross-modal attention weights, and use it to evaluate the contribution of each modality parameter to the task;

[0008] S3: Employs a structured pruning strategy, dynamically adjusting the pruning ratio based on importance scores to retain key visual channels and text neurons, thereby achieving a lightweight model;

[0009] S4: Fine-tuning through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model, and a loss function including cross-modal contrastive loss is used to enhance modality alignment;

[0010] S5: Implement targeted fine-tuning, and optimize multi-objective loss for various text tasks with heterogeneous texts;

[0011] The process involves acquiring and preprocessing bimodal data, which includes both visual and textual modal data. The visual modal data consists of construction site monitoring images, while the textual modal data comprises three types of heterogeneous text: on-site standards. Equipment labels and types of hidden dangers ;

[0012] Visual modality preprocessing converts the input construction site monitoring images into visual feature vectors using PatchEmbed and ViT:

[0013] ;

[0014] In the formula, For monitoring images, Global visual features are extracted using a Transformer encoder; The image is segmented into patches and converted into a vector sequence;

[0015] Preprocessing of text modal data includes: processing long texts in the heterogeneous text. The BERT encoder is used to extract semantic features for short texts. The BiLSTM sequence model is used to capture sequence features for structured text. The embedding layer is used to map the data into vectors; the comprehensive representation is generated through a feature concatenation formula, the expression of which is:

[0016] ) );

[0017] In the formula, This is a long text. For short text, For structured text; For the BERT model to handle long texts that meet security standards, A BiLSTM model for processing short text device labels. An embedding model for processing structured text of potential hazard types;

[0018] The joint importance function is calculated as follows:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] In the formula, The vector feature representation of image data, To fuse the vector feature representations of the three types of text; , To reduce the parameters of vertical visual and text models, Losses due to construction site tasks This is the cross-modal attention weight matrix. This is the balance coefficient.

[0024] In the preferred embodiment, the pruning threshold is extended to the multimodal level by jointly evaluating visual and text thresholds; if the importance score involves cross-modal interaction, the pruning ratio is adjusted to balance modal contributions; secondary verification is performed on the retained key visual channels to ensure alignment with the text neurons; the pruning ratio is calculated using the function to generate a compressed model framework; and the pruning results are integrated into the student model.

[0025] In a preferred embodiment, the structured pruning strategy includes: determining a pruning threshold based on the importance score, wherein a visual threshold is set for the visual modality and a text threshold is set for the text modality; and generating a mask by combining the thresholds; dynamically adjusting the pruning ratio, wherein the pruning ratio is expressed as:

[0026] ;

[0027] In the formula, The pruning ratio for visual branches. The pruning ratio for text branches. For variance, The mean, , A set of importance scores for visual and text models;

[0028] By preserving important visual channels and text neurons through channel-level pruning, the mask expression is:

[0029] ;

[0030] ;

[0031] In the formula, For the binary mask of the i-th parameter in the visual or text branch, Score the importance of the i-th parameter in the visual or text branch. The pruning threshold for visual branches. The pruning threshold for text branches. Standard deviation, The mean;

[0032] If the importance score is below the threshold, the corresponding parameter is removed to compress the model parameters; the performance of the pruned model is verified, and parameters that significantly contribute to the task are retained through iterative adjustments.

[0033] In the preferred embodiment, the fine-tuning through knowledge distillation includes: using the original bimodal model as the teacher model and the pruned model as the student model, defining the loss function as follows:

[0034] ;

[0035] In the formula, For cross-modal contrast loss, These are the weighting coefficients. Cross-entropy loss measures the difference between the student model's predictions and the teacher model's predictions. For student model prediction, For teacher model prediction, For relative entropy loss, For the student model probability distribution, The probability distribution for the teacher model;

[0036] The cross-modal contrast loss enhances the modal alignment between visual and textual elements; the temperature parameter softens the probability distribution of the teacher model, guiding the student model's learning decision-making logic; and after optimizing the weight coefficients, the loss function is minimized, enabling fine-tuning of the student model.

[0037] In the preferred embodiment, fine-tuning through knowledge distillation includes: initializing the weights of the teacher model and the student model and aligning them based on pre-training; if the cross-modal contrastive loss calculation involves feature similarity, then optimizing the modal association through the loss function; adjusting the temperature parameter to adapt to the softening requirements of the probability distribution; fusing the task loss and contrastive loss to generate a fine-tuning path; evaluating the learning performance of the student model and minimizing the total loss through iteration.

[0038] In the preferred embodiment, the targeted fine-tuning includes: designing the multi-objective loss for the multiple text tasks, wherein the multi-objective loss expression is:

[0039] ;

[0040] In the formula, These are the weighting coefficients for safety standard tasks, equipment identification tasks, and hazard classification tasks, respectively. To focus on the loss function; For safety standard testing output and labeling, It is a binary classification loss function; For multi-label classification loss, and For device identification output and tags, and Classify and label potential hazards.

[0041] Secondly, the present invention provides a dual-modal scene lightweight vertical category model construction system, applicable to the aforementioned dual-modal scene lightweight vertical category model construction method, comprising:

[0042] The bimodal data module is used to acquire and preprocess bimodal data, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in a hierarchical manner to form a comprehensive representation.

[0043] The importance scoring module is used to perform visual-text cross-modal importance scoring. It defines a joint importance function that fuses gradient information and cross-modal attention weights to evaluate the contribution of each modality parameter to the task.

[0044] The structured pruning module is used to adopt a structured pruning strategy, dynamically adjust the pruning ratio based on the importance score, retain key visual channels and text neurons, and achieve model lightweighting.

[0045] The distillation fine-tuning module is used to fine-tune through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model, and a loss function including cross-modal contrastive loss is used to enhance modality alignment;

[0046] The targeted fine-tuning module is used to perform targeted fine-tuning, designing multi-objective losses for various text tasks to optimize the model and improve its adaptability in specific scenarios.

[0047] Thirdly, the present invention provides an electronic device, including a memory and a processor;

[0048] The memory is used to store computer programs;

[0049] The processor is configured to implement the bimodal scene lightweight vertical category model construction method when executing the computer program.

[0050] This invention provides a lightweight vertical class model construction method for bimodal pruning training. It acquires and preprocesses bimodal data, then performs visual-text cross-modal importance scoring. A structured pruning strategy is employed, dynamically adjusting the pruning ratio based on the importance score to retain key visual channels and text neurons, thus achieving model lightweighting. After knowledge distillation fine-tuning, targeted fine-tuning is implemented, and multi-objective losses are designed for optimization across multiple text tasks. By extracting and concatenating features hierarchically to form a comprehensive representation, the value of textual information is fully explored, improving the efficiency of text modality utilization. A loss function including cross-modal contrastive loss is used to enhance modality alignment, improve detection accuracy, and enhance the model's adaptability in specific scenarios. Attached Figure Description

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0052] Figure 1 This is a schematic diagram illustrating the overall implementation of the method of this invention;

[0053] Figure 2 This is a flowchart of the construction method of the present invention. Detailed Implementation

[0054] Example 1

[0055] like Figure 1-2 As shown, a method for constructing a lightweight vertical class model with bimodal pruning training includes the following steps:

[0056] S1: Acquire dual-modal data and preprocess it, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in layers to form a comprehensive representation.

[0057] S2: Perform visual-text cross-modal importance scoring, define a joint importance function that integrates gradient information and cross-modal attention weights, and use it to evaluate the contribution of each modality parameter to the task.

[0058] S3: A structured pruning strategy is adopted, which dynamically adjusts the pruning ratio based on the importance score, retains key visual channels and text neurons, and achieves model lightweighting.

[0059] S4: Fine-tuning through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model, and modality alignment is enhanced using a loss function that includes cross-modal contrastive loss.

[0060] S5: Implement targeted fine-tuning, design multi-objective loss for various text tasks and optimize it to improve the model's adaptability in specific scenarios.

[0061] This embodiment acquires and preprocesses bimodal data, then performs visual-text cross-modal importance scoring. A structured pruning strategy is employed, dynamically adjusting the pruning ratio based on the importance score to retain key visual channels and text neurons, achieving a lightweight model. After knowledge distillation fine-tuning, targeted fine-tuning is implemented, and multi-objective losses are designed for optimization across various text tasks. By extracting and concatenating features hierarchically to form a comprehensive representation, the value of textual information is fully explored, improving the efficiency of text modality utilization. A loss function including cross-modal contrastive loss is used to enhance modality alignment, improving detection accuracy and enhancing the model's adaptability in specific scenarios.

[0062] In step S1, bimodal data is acquired and preprocessed. Acquiring bimodal data involves image data collected from the monitoring system and associated text information of various types. The bimodal data includes two categories: visual modality and text modality. The visual modality consists of construction site monitoring images, while the text modality includes three types of heterogeneous text, including on-site standards. Equipment labels and types of hidden dangers .

[0063] Surveillance images typically contain target objects, environmental background, and spatiotemporal information within a scene. After preprocessing, the image data is transformed into standardized visual feature representations. Text modal data, on the other hand, originates from multiple heterogeneous data sources, including long text descriptions, short text tags, and structured text records. This textual information provides semantic supplementation and contextual interpretation for the image content.

[0064] S11: For long texts in heterogeneous texts, an encoder is used to extract semantic features; for short texts, a sequence model is used to capture sequence features; and for structured texts, an embedding layer is used to map them into vectors.

[0065] Specifically, long text processing employs a transformer-based encoder model: capturing deep semantic relationships within the text through a multi-layered self-attention mechanism. Short text sequence models utilize recurrent neural network structures to process labeled text information. Since short texts typically contain keywords and concise descriptions, the sequence model controls information flow through gating mechanisms, selectively retaining and forgetting historical information. Structured text processing employs a specially designed embedding layer architecture. Structured text typically contains classification labels, numerical attributes, and hierarchical information, all of which possess a clear semantic structure and fixed formatting specifications.

[0066] S12: Long text, short text, and structured text are combined using a feature concatenation formula to generate a comprehensive representation, the expression of which is:

[0067] ) );

[0068] In the formula, For long text (security standard). For short text (device label). For structured text (hazard type); A BERT model for handling long texts that meet security standards; A BiLSTM model for processing short text of device labels; An embedding model for handling structured texts of potential hazards.

[0069] Visual modality preprocessing converts the input construction site monitoring images into visual feature vectors using PatchEmbed and ViT:

[0070] ;

[0071] In the formula, For monitoring images, Global visual features are extracted using a Transformer encoder; The image is segmented into patches and converted into a vector sequence.

[0072] In this embodiment, the feature concatenation process needs to ensure compatibility of different modal features in terms of dimension and scale. (Long text features) Short text features typically have high dimensionality, containing rich semantic information and contextual relationships; their feature vector lengths can reach hundreds of dimensions. While relatively low in dimensionality, it contains key identifying information and classification attributes. Structured text features. It has a fixed dimensional structure, with each dimension corresponding to a specific attribute field. The concatenation operation combines three different types of text features into a unified comprehensive representation vector by concatenating them along the feature dimensions.

[0073] Before feature concatenation, normalization is required to ensure that the numerical ranges of features from different sources remain consistent.

[0074] S13: Perform preliminary alignment between the comprehensive representation and visual modal features to form input for subsequent scoring.

[0075] Visual modal features are extracted from surveillance images using a convolutional neural network. Each convolutional layer contains multiple convolutional kernels, which perform convolution operations on the input feature map using a sliding window operation to generate a corresponding feature response map. Pooling layers downsample the feature map, reducing spatial dimensionality while retaining important feature information.

[0076] The initial alignment of visual features and text features is achieved by combining linear transformation and attention mechanisms.

[0077] S14. If the dimensions of heterogeneous texts do not match, the feature dimensions are unified through a dimension adjustment mechanism to ensure the stability of the comprehensive representation after splicing.

[0078] Dimensionality adjustment mechanisms employ various strategies to address the issue of inconsistent feature dimensions. When the dimension of a text feature is lower than the target dimension, the feature vector is expanded using zero-padding or repeated padding. Zero-padding adds zero-valued elements to the end of the feature vector, preserving the original information while reaching the target dimension. Repeated padding expands the dimension by cyclically copying existing feature elements, a method that maintains the statistical properties of the features. When the feature dimension is higher than the target dimension, principal component analysis or linear projection is used for dimensionality reduction.

[0079] Principal component analysis (PCA) identifies the main directions of change in data by calculating the eigenvalues ​​and eigenvectors of the feature covariance matrix. Retaining the principal components corresponding to the first few largest eigenvalues ​​allows for the preservation of original information to the greatest extent possible while reducing dimensionality. Linear projection, on the other hand, learns a transformation matrix to map high-dimensional features to a low-dimensional space. This transformation matrix is ​​optimized during training using backpropagation to ensure that the dimensionality-reduced features still effectively support downstream tasks. The dimensionality-adjusted feature vectors maintain both numerical stability and semantic integrity.

[0080] In step S2, cross-modal importance scoring is a crucial step in the model pruning process. It uses quantitative analysis to determine the contribution of each model parameter to the final task performance. Importance scoring considers not only the importance of parameters within a single modality but also the interactive and synergistic effects between different modalities, thus identifying parameters that truly play a key role in cross-modal tasks.

[0081] Step S21: Calculate the joint importance function, where the expression for the joint importance function is:

[0082] ;

[0083] ;

[0084] ;

[0085]

[0086] In the formula, The vector feature representation of image data. To fuse the vector feature representations of the three types of text; , To reduce the parameters of vertical visual and text models, Losses due to construction site tasks This is the cross-modal attention weight matrix. This is the balance coefficient.

[0087] In this embodiment, the joint importance function evaluates parameter importance by weighted combination of two key metrics. The cross-modal attention weight matrix A captures the interdependence between visual and text features. The attention weights are calculated using a query-key-value matching mechanism, with visual features as the query vector and text features as key-value pairs, and similarity scores are calculated through dot product operations. The similarity scores are normalized using a soft maximization function to form the attention weights, reflecting the degree of attention paid by visual features to different text features.

[0088] The balancing coefficient λ needs to be adjusted based on the specific task characteristics and data features. When λ is close to 1, the importance score mainly relies on gradient information, focusing more on the direct impact of parameters on the task loss. When λ is close to 0, the score relies more on cross-modal attention weights, emphasizing the importance of interactions between modalities.

[0089] In practical applications, the optimal λ value is determined by methods such as grid search or Bayesian optimization, so that the importance score can accurately reflect the true contribution of the parameter.

[0090] Step S22: Obtain the contribution of each parameter from the task loss based on the gradient information.

[0091] The calculation of gradient information involves differential analysis of the parameter space of the entire neural network. The magnitude of the absolute value of the gradient is directly related to the importance of the parameter. The larger the absolute value of the parameter, the more significant its impact on the model performance.

[0092] The calculation of parameter contribution also considers historical information and trends in gradients. By maintaining an exponential moving average of the gradient, a more stable and reliable importance estimate can be obtained.

[0093] Step S23: Evaluate the interaction between the visual channel and the text neurons using a cross-modal attention weight matrix.

[0094] The construction of the cross-modal attention weight matrix is ​​based on semantic association analysis between visual and textual features. The evaluation of mutual influence employs a bidirectional attention mechanism. Forward attention calculates the degree of attention visual features give to textual features, while backward attention calculates the degree of attention textual features give to visual features. The combination of bidirectional attention can more comprehensively capture the interdependencies between the two modalities.

[0095] S24: If the balance coefficients are adjusted experimentally, the gradient information and the cross-modal attention weight matrix are fused to obtain the final importance score for guiding pruning.

[0096] In this embodiment, the experimental adjustment process of the balance coefficient involves multiple rounds of validation and performance evaluation. First, the model's performance is tested on the validation set under different λ values, recording key metrics such as accuracy, recall, and F1 score. Then, the trends of these metrics with varying λ values ​​are analyzed to identify the optimal parameter range. After determining the optimal λ value, this parameter is used to calculate the importance score of all model parameters, forming a score vector to guide pruning decisions; the fused importance score exhibits better robustness and accuracy.

[0097] Step S25: Calculate the importance score for each visual channel and text neuron to form the basis for evaluation.

[0098] The calculation of visual channel importance score needs to consider the combined contribution of all parameters within the channel. Each visual channel contains multiple convolution kernel parameters and bias parameters, which determine the feature extraction capability of that channel.

[0099] Step S3 involves a structured pruning strategy that reduces model complexity by systematically removing entire structural units. Compared to unstructured pruning, structured pruning removes complete channels, layers, or neurons, rather than individual weight parameters. This pruning method maintains the regularity of the model structure, facilitating efficient inference on various hardware platforms. The structured pruned model does not require special sparse computing libraries and can be directly deployed and run using standard deep learning frameworks.

[0100] S31: Extend the pruning threshold to the multimodal level by jointly evaluating visual and text thresholds; if the importance score involves cross-modal interaction, adjust the pruning ratio to balance modal contributions; perform secondary validation on the retained key visual channels to ensure alignment with text neurons; calculate the pruning ratio using a function to generate a compressed model framework; integrate the pruning results into the student model for subsequent distillation.

[0101] S32: A structured pruning strategy is adopted, including: determining the pruning threshold based on the importance score, wherein a visual threshold is set for the visual modality and a text threshold is set for the text modality.

[0102] S33: Dynamically adjust the pruning ratio, where the pruning ratio is expressed as:

[0103] ;

[0104] In the formula, The pruning ratio for visual branches. The pruning ratio for text branches. For variance, The mean, , A set of importance scores for visual and text models.

[0105] S34: Important visual channels and text neurons are preserved through channel-level pruning. The masking expression is:

[0106] ;

[0107] ;

[0108] In the formula, For the binary mask of the i-th parameter in the visual or text branch, Score the importance of the i-th parameter in the visual or text branch. The pruning threshold for visual branches. The pruning threshold for text branches. Standard deviation, This is the mean.

[0109] S35: If the importance score is below the threshold, remove the corresponding parameter to compress the model parameters.

[0110] S36: Verify the performance of the pruned model and ensure that parameters that contribute significantly to the task are retained through iterative adjustments.

[0111] In step S4, knowledge distillation preserves model performance by having a lightweight student model learn the knowledge representations of the original teacher model. The teacher model contains a complete network structure and parameters, possessing powerful feature extraction and pattern recognition capabilities. Although the student model has a simplified structure, it can achieve near-teacher model performance at a smaller model size by mimicking the output distribution and intermediate feature representations of the teacher model. The knowledge distillation process not only transmits the final prediction result but also includes deep knowledge such as intermediate layer features, attention weights, and activation patterns.

[0112] S41: Fine-tuning through knowledge distillation, including: initializing the weights of the teacher and student models and aligning them based on pre-training.

[0113] S42: Define the loss function, where the expression for the loss function is:

[0114] ;

[0115] In the formula, For cross-modal contrast loss, These are the weighting coefficients. Cross-entropy loss measures the difference between the student model's predictions and the teacher model's predictions. For student model prediction, For teacher model prediction; The relative entropy loss measures the difference in probability distribution between the student model and the teacher model. For the student model probability distribution, This represents the probability distribution of the teacher model.

[0116] In this embodiment, task loss measures the performance of the student model on a specific task. For classification tasks, the task loss typically employs the cross-entropy loss function, calculating the difference between the student model's predicted distribution and the true labels. For regression tasks, loss functions such as mean squared error or mean absolute error are used. Task loss ensures that the student model can correctly complete the target task and is a fundamental constraint in the knowledge distillation process.

[0117] S43: Enhance visual and textual modal alignment through cross-modal contrastive loss; soften the probability distribution of the teacher model using temperature parameters to guide the student model in learning decision-making logic; minimize the loss function if the weight coefficients are optimized to achieve fine-tuning of the student model.

[0118] S44: If cross-modal contrastive loss calculation involves feature similarity, then optimize modal association through loss function; adjust temperature parameters to adapt to the softening requirements of probability distribution; fuse task loss and contrastive loss to generate fine-tuned path.

[0119] S45: If the weight coefficients are optimized, the loss function is minimized to fine-tune the student model. The learning performance of the student model is evaluated by iteratively minimizing the total loss.

[0120] In step S5, targeted fine-tuning is a process of optimizing the model specifically for a particular application scenario. Unlike general model training, targeted fine-tuning focuses on the model's performance in the target scenario, adjusting model parameters to adapt to specific data distributions and task requirements. This process requires in-depth analysis of the characteristics of the target scenario, identification of key performance indicators and constraints, and then designing corresponding optimization strategies.

[0121] S51: Implement targeted fine-tuning, including: designing multi-objective loss for multiple text tasks, where the multi-objective loss expression is:

[0122] ;

[0123] In the formula, These are the weight coefficients for safety standard tasks, equipment identification tasks, and hazard classification tasks, respectively. To focus on the loss function and address the class imbalance problem; For safety standard testing output and labeling.

[0124] This is a binary classification loss function used for device identification tasks. For device identification output and tags, This is the loss for multi-label classification, used for hazard classification tasks; Classify and label potential hazards.

[0125] S52: Allocate resources based on task weight to improve adaptability to specific scenarios.

[0126] S53: Multi-stage optimization is used to process multi-objective losses, ensuring improved model accuracy.

[0127] S54: If multiple text tasks include compliance detection and classification, then integrate the comprehensive representation for targeted adjustments.

[0128] S55: Verify the recall rate of the fine-tuned model in complex scenarios to form the final lightweight model.

[0129] In this embodiment, recall verification employs a tiered testing strategy, classifying complex scenarios according to difficulty levels. The basic level includes common scenarios under standard conditions, the medium level involves the influence of some interfering factors, and the high level simulates challenging scenarios under extreme conditions. By evaluating performance at different levels, the robustness and adaptability of the model can be comprehensively assessed. Ultimately, the lightweight model needs to achieve preset performance standards across all test levels to ensure stable and reliable operation in actual deployments.

[0130] Let's take personnel safety compliance testing as an example to illustrate:

[0131] 1. Data Preparation

[0132] This embodiment collects two types of core data to construct the training set: 1) Visual data comes from 15,000 frames of surveillance video from a large construction site, covering different time periods, weather conditions, and lighting conditions. Each frame is labeled with key safety elements such as the wearing status of safety helmets and the integrity of guardrails. 2) Text data contains three types of heterogeneous information: First, a complete text library of the "Construction Safety Inspection Standard" (JGJ59-2011), containing more than 1,200 safety regulations; second, label data of 127 common types of equipment on the construction site, such as structured identifiers like "tower crane - model QTZ80" and "construction elevator - SC200 / 200"; and third, descriptive text of 20 typical safety hazards, including standardized expressions such as "no safety belt for working at heights," "lack of edge protection," and "blocked fire exits." All text data has been cleaned and standardized to ensure an effective correspondence with the visual scene.

[0133] 2. ViT-Base was used as the architecture for the visual branch, with an initial parameter count of 86M; BERT-Base was used as the architecture for the text branch, with an initial parameter count of 110M. During the pruning phase, key parameters were set as follows: cross-modal balance coefficient λ = 0.4, and visual branch importance threshold. =0.75, text branch importance threshold =0.68. The knowledge distillation stage uses task loss weights with α=0.8 and contrastive loss weights with β=0.2, with a distillation temperature parameter T=4 and a softening probability distribution. After model initialization, cross-modal alignment training is first performed based on the pre-trained weights to establish a semantic connection between visual and textual features.

[0134] 3. Training Process

[0135] The training process employs a multi-stage optimization strategy. The first stage involves cross-modal feature extraction: the visual branch segments the image into 16×16 patches using a ViT encoder, generating a 768-dimensional feature vector through positional encoding and self-attention; the text branch uses a hierarchical processing mechanism: semantic features of safety standard text are extracted using a BERT encoder, sequence features of device label text are captured using a bidirectional LSTM network, and hazard type text is mapped to a fixed-dimensional vector through an embedding layer. These three types of text features are concatenated to form a 2304-dimensional comprehensive representation. The second stage performs joint importance assessment: the gradient contribution of each visual channel and text neuron is calculated, and an importance score is generated by combining cross-modal attention weights. The visual branch score includes gradient information and text attention weights, while the text branch score includes gradient information and visual attention weights. The third stage implements structured pruning: the pruning ratio is dynamically determined based on the importance score, improving the average pruning rate of both the visual and text branches, and extracting parameters that significantly contribute to the safety detection task. The fourth stage involves knowledge distillation: using the original bimodal large model as the teacher and the pruned model as the student, the student model learns the decision-making logic and modal association patterns of the teacher model by minimizing cross-entropy loss, KL divergence loss, and cross-modal contrastive loss. The fifth stage implements targeted fine-tuning: for three tasks—safety standard compliance, equipment identification, and hazard classification—multi-objective optimization is performed using a weight ratio of 0.5:0.3:0.2, focusing on improving the model's adaptability to specific safety scenarios on construction sites.

[0136] 4. Deployment Results: The trained model was deployed on a Jetson Nano edge computing platform (4GB RAM). In a real-world construction environment, the system achieved a real-time processing speed of 16FPS with an input image of 640×640 resolution, reducing inference latency, improving the accuracy of safety violation detection, and increasing the accuracy of text standard compliance detection; it also improved the system's recall rate in complex scenarios such as occlusion and low light.

[0137] The technical effects achieved by this embodiment are as follows:

[0138] In terms of model lightweighting, the number of parameters of the original multimodal large model is greatly reduced by the structured pruning strategy, which enables the model to be deployed on resource-constrained edge devices, such as field end computers or station cameras, while general large models are difficult to deploy in this way due to their large number of parameters.

[0139] In terms of real-time processing capabilities, inference latency has been reduced. Tests on the Jetson Nano edge device show that both model inference time and inference speed meet the needs of real-time monitoring of construction sites, enabling the system to respond promptly to safety violations and providing effective technical support for on-site safety management.

[0140] The hierarchical feature extraction mechanism significantly improves the processing efficiency of three types of text data: By differentiating heterogeneous texts, BERT is used to encode long texts of security standards, BiLSTM is used to process short texts of device labels, and the embedding layer maps structured texts of potential hazards, so that the text information can be fully mined and effectively utilized.

[0141] From an engineering application perspective, it significantly reduces deployment costs and is suitable for resource-constrained and complex environments such as construction sites.

[0142] Example 2

[0143] To further illustrate with reference to Embodiment 1, a dual-modal lightweight vertical category model construction system for scenes, applicable to the dual-modal lightweight vertical category model construction method of Embodiment 1, includes:

[0144] The bimodal data module is used to acquire and preprocess bimodal data, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in a hierarchical manner to form a comprehensive representation.

[0145] The importance scoring module is used to perform visual-text cross-modal importance scoring. It defines a joint importance function that integrates gradient information and cross-modal attention weights to evaluate the contribution of each modality parameter to the task.

[0146] The structured pruning module is used to adopt a structured pruning strategy, dynamically adjust the pruning ratio based on importance scores, retain key visual channels and text neurons, and achieve model lightweighting.

[0147] The distillation fine-tuning module is used to fine-tune the model through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model. A loss function that includes cross-modal contrastive loss is used to enhance modality alignment.

[0148] The targeted fine-tuning module is used to perform targeted fine-tuning, designing multi-objective losses for various text tasks to optimize the model and improve its adaptability in specific scenarios.

[0149] This embodiment provides a method for constructing a lightweight vertical class model for dual-modal pruning training. The working process, working details, and technical effects are described in Embodiment 1, and will not be repeated here.

[0150] Example 3

[0151] This application provides an electronic device, including a memory and a processor.

[0152] Memory is used to store computer programs.

[0153] The processor is used to implement, when executing a computer program, a method for constructing a lightweight vertical class model for bimodal pruning training according to Embodiment 1, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0154] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO), and / or last-in-first-out memory (FILO), etc.; the processor may be limited to microprocessors of the STM32F105 series, ARM (Advanced RISC Machines), x86 architecture processors, or processors with integrated NPU (neural-network processing units).

[0155] In this embodiment, a computer-readable storage medium, i.e., a computer-readable storage medium storing instructions, executes a method for constructing a lightweight vertical category model for bimodal pruning training as described in the first aspect when the instructions are run on a computer. The computer-readable storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0156] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for constructing a lightweight vertical category model for a dual-modal scene, characterized in that, Includes the following steps: S1: Acquire dual-modal data and preprocess it, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in layers to form a comprehensive representation. S2: Perform visual-text cross-modal importance scoring, define a joint importance function that integrates gradient information and cross-modal attention weights, and use it to evaluate the contribution of each modality parameter to the task; S3: Employs a structured pruning strategy, dynamically adjusting the pruning ratio based on importance scores to retain key visual channels and text neurons, thereby achieving a lightweight model; S4: Fine-tuning through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model, and a loss function including cross-modal contrastive loss is used to enhance modality alignment; S5: Implement targeted fine-tuning, and optimize multi-objective loss for various text tasks with heterogeneous texts; The process involves acquiring and preprocessing bimodal data, which includes both visual and textual modal data. The visual modal data consists of construction site monitoring images, while the textual modal data comprises three types of heterogeneous text: on-site standards. Equipment labels and types of hidden dangers ; Visual modality preprocessing converts the input construction site monitoring images into visual feature vectors using PatchEmbed and ViT: ； In the formula, For monitoring images, Global visual features are extracted using a Transformer encoder; The image is segmented into patches and converted into a vector sequence; Preprocessing of text modal data includes: processing long texts in the heterogeneous text. The BERT encoder is used to extract semantic features for short texts. The BiLSTM sequence model is used to capture sequence features for structured text. The embedding layer is used to map the vector; The comprehensive representation is generated using a feature concatenation formula, expressed as follows: ) )； In the formula, This is a long text. For short text, For structured text; For the BERT model to handle long texts that meet security standards, A BiLSTM model for processing short text device labels. An embedding model for processing structured text of potential hazard types; The joint importance function is calculated as follows: ； ； ； ； In the formula, The vector feature representation of image data. To fuse the vector feature representations of the three types of text; , To reduce the parameters of vertical visual and text models, Losses due to construction site tasks This is the cross-modal attention weight matrix. This is the balance coefficient.

2. The method for constructing a lightweight vertical category model for a dual-modal scene according to claim 1, characterized in that, The pruning threshold is extended to the multimodal level by jointly evaluating visual and text thresholds; if the importance score involves cross-modal interaction, the pruning ratio is adjusted to balance modal contributions; secondary validation is performed on the retained key visual channels to ensure alignment with the text neurons; The pruning ratio is calculated using the function to generate a compressed model framework; the pruning results are then integrated into the student model.

3. The method for constructing a lightweight vertical category model for a dual-modal scene according to claim 1, characterized in that, The structured pruning strategy includes: determining pruning thresholds based on the importance score, wherein a visual threshold is set for the visual modality and a text threshold is set for the text modality; and generating a mask by combining the thresholds; dynamically adjusting the pruning ratio, wherein the pruning ratio is expressed as: ； In the formula, The pruning ratio for visual branches. The pruning ratio for text branches. For variance, The mean, , A set of importance scores for visual and text models; By preserving important visual channels and text neurons through channel-level pruning, the mask expression is: ； ； In the formula, For the binary mask of the i-th parameter in the visual or text branch, Score the importance of the i-th parameter in the visual or text branch. The pruning threshold for visual branches. The pruning threshold for text branches. Standard deviation, The mean; If the importance score is below the threshold, the corresponding parameter is removed to compress the model parameters; the performance of the pruned model is verified, and parameters that significantly contribute to the task are retained through iterative adjustments.

4. The method for constructing a lightweight vertical category model for a dual-modal scene according to claim 1, characterized in that, The fine-tuning through knowledge distillation includes: using the original bimodal model as the teacher model and the pruned model as the student model, defining the loss function with the following expression: ； In the formula, For cross-modal contrast loss, These are the weighting coefficients. Cross-entropy loss measures the difference between the student model's predictions and the teacher model's predictions. For student model prediction, For teacher model prediction, For relative entropy loss, For the student model probability distribution, The probability distribution for the teacher model; The cross-modal contrast loss enhances the modal alignment between visual and textual elements; the temperature parameter softens the probability distribution of the teacher model, guiding the student model's learning decision-making logic; and after optimizing the weight coefficients, the loss function is minimized, enabling fine-tuning of the student model.

5. The method for constructing a lightweight vertical category model for a dual-modal scene according to claim 4, characterized in that, The fine-tuning process via knowledge distillation includes: initializing the weights of the teacher and student models and aligning them based on pre-training; optimizing modal associations through the loss function if the cross-modal contrastive loss calculation involves feature similarity; adjusting the temperature parameter to accommodate the softening requirements of the probability distribution; fusing the task loss and contrastive loss to generate a fine-tuning path; and evaluating the learning performance of the student model by iteratively minimizing the total loss.

6. The method for constructing a lightweight vertical category model for a dual-modal scene according to claim 1, characterized in that, The implementation of targeted fine-tuning includes: designing the multi-objective loss for the multiple types of text tasks, wherein the multi-objective loss expression is: ； In the formula, These are the weighting coefficients for safety standard tasks, equipment identification tasks, and hazard classification tasks, respectively. To focus on the loss function; and For safety standard testing output and labeling, It is a binary classification loss function; For multi-label classification loss, and For device identification output and tags, and Classify and label potential hazards.

7. A dual-modal lightweight vertical category model construction system, characterized in that, A method for constructing a lightweight vertical category model for a bimodal scene as described in any one of claims 1-6, comprising: The bimodal data module is used to acquire and preprocess bimodal data, where the visual modality is surveillance images and the text modality is heterogeneous text. Features are extracted and spliced ​​in a hierarchical manner to form a comprehensive representation. The importance scoring module is used to perform visual-text cross-modal importance scoring. It defines a joint importance function that fuses gradient information and cross-modal attention weights to evaluate the contribution of each modality parameter to the task. The structured pruning module is used to adopt a structured pruning strategy, dynamically adjust the pruning ratio based on the importance score, retain key visual channels and text neurons, and achieve model lightweighting. The distillation fine-tuning module is used to fine-tune through knowledge distillation, where the teacher model is the original bimodal model and the student model is the pruned model, and a loss function including cross-modal contrastive loss is used to enhance modality alignment; The targeted fine-tuning module is used to perform targeted fine-tuning, designing multi-objective losses for various text tasks to optimize the model and improve its adaptability in specific scenarios.

8. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement, when executing the computer program, a dual-modal scene lightweight vertical category model construction method as described in any one of claims 1 to 6.

Images