A multi-modal large model light distillation optimization method and system for oilfield edge deployment

Abstract

The application provides a multimodal large model light distillation optimization method and system for oil field edge deployment, and belongs to the field of oil field inspection. The method solves the problem that the existing model light distillation technology does not match the actual needs of the oil field edge in key links such as architecture adaptation, data supply, and training controllability. The method comprises the following steps: obtaining and processing oil field multimodal data; constructing a teacher model and a student model based on the oil field multimodal data; transferring the knowledge of the teacher model to the student model based on knowledge distillation; sequentially performing convolution layer structural pruning and dynamic quantization processing on the student model trained by knowledge distillation to obtain a light model; and converting the light model into a general format model file suitable for an edge inference engine. The method is mainly used in the field of oil and gas storage area security.

Description

Technical Field

[0001] This invention belongs to the field of oilfield inspection, and in particular relates to a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield. Background Technology

[0002] In the process of digital transformation and intelligent upgrading of oilfields, multimodal large models have shown great application potential in scenarios such as oilfield inspection, well site equipment status monitoring, and oil and gas storage area security because they can integrate and analyze various information such as images, videos, and various sensor data. These models are usually trained on cloud servers and have powerful feature extraction and complex pattern recognition capabilities. However, the oilfield production site environment is special, with many deployment points and extremely high requirements for real-time performance and reliability. Therefore, the ideal deployment location is at the source of data generation, i.e., the oilfield edge, such as embedded industrial control computers, low-power AI boards, and other devices. However, multimodal large models trained in the cloud generally have complex architectures, a large number of parameters, and a huge amount of computation. This creates a sharp contradiction with the hardware constraints of oilfield edge devices, such as limited computing power, small memory (GPU memory), and scarce storage resources. As a result, high-performance cloud models cannot be directly deployed to the edge for real-time inference.

[0003] Currently, the industry is attempting to address the challenges of deploying large models at the edge using lightweight technologies such as model compression, knowledge distillation, and quantization. However, these general solutions reveal a series of insurmountable flaws when faced with the specific scenario of oil fields: First, regarding model architecture and hardware adaptation, existing large multimodal models are primarily designed for high-performance GPUs in the cloud. Their model depth, width, and attention mechanisms introduce significant computational overhead. When attempting to deploy them to edge devices in oil fields with computing power limited to trillions of operations per second, severe inference latency occurs, failing to meet the stringent real-time requirements of scenarios such as oil field inspections and equipment fault warnings. Current technologies lack lightweight multimodal model architectures specifically designed for low-computing-power hardware at the oil field edge. Simple network scaling, such as uniformly reducing the number of channels, often severely impairs the model's ability to extract specific multimodal data features from oil fields.

[0004] Secondly, there are significant bottlenecks in acquiring multimodal data from oilfields at the model training data level. High-quality oilfield inspection images and equipment operation sensor data are not only limited by safety regulations, equipment status, and geographical environment, resulting in a small total sample size, but also require domain expert intervention for detailed labeling of normal, minor anomalies, and severe anomalies, which is costly and time-consuming. Existing lightweight model methods typically assume sufficient and well-labeled training data, which is not applicable in oilfield scenarios. Simply relying on limited, potentially unevenly distributed real data for training and distillation results in models with poor generalization ability, making it difficult to adapt to different oilfield blocks and equipment models, and easily leading to false alarms and missed alarms.

[0005] Furthermore, in terms of lightweight training processes and control, existing techniques such as knowledge distillation exhibit rigidity when applied to multimodal oilfield data. For example, the key hyperparameter distillation temperature in knowledge distillation is typically set to a fixed value to control the smoothness of the soft labels output by the teacher model. However, the image modalities in oilfield scenarios are rich in spatial texture, and the time-series numerical values ​​of the sensor modalities are heterogeneous. The correlation (relevance) between their data features varies depending on the specific working conditions and equipment type. A fixed distillation temperature cannot adapt to this variation, resulting in low knowledge transfer efficiency. When the correlation is low, smoother soft labels are needed to promote cross-modal knowledge transfer, while when the correlation is high, sharper labels are needed to focus on classification accuracy. Similarly, the weight parameters used to balance distillation loss and classification loss lack a dynamic adjustment mechanism, making the lightweighting process uncontrollable and difficult to maintain high accuracy stably while compressing the model.

[0006] Finally, at the model deployment and engineering level, there are compatibility barriers between the model file formats generated by cloud-based training frameworks and the diverse edge inference engines used in oilfields. Current technologies lack an automated process for format conversion and optimization of lightweight models. Deployment engineers often need to perform tedious operator rewriting, graph optimization, and manual quantization calibration for specific hardware. This process is technically demanding, labor-intensive, and prone to introducing errors, resulting in long and inefficient cycles from model training to actual deployment, severely hindering the practical application of the technology. Summary of the Invention

[0007] In view of this, the present invention aims to propose a lightweight distillation optimization method and system for multimodal large models deployed at the edge of oil fields, in order to solve the problem that existing lightweight model technologies are not compatible with the actual needs of oil field edge deployments in key aspects such as architecture adaptation, data supply, and training controllability.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal large-scale lightweight distillation optimization method for deployment at the edge of oil fields, the method comprising: Acquire and process multimodal data from the oilfield, including image data and sensor data; A teacher model and a student model based on oilfield multimodal data are constructed. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. Knowledge distillation is used to transfer knowledge from the teacher model to the student model. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The student model trained by knowledge distillation is subjected to convolutional layer structured pruning and dynamic quantization in sequence to obtain a lightweight model. The lightweight model is converted into a common format model file suitable for edge inference engines.

[0009] Furthermore, a preferred method is proposed, wherein the oilfield multimodal data is a mixture of synthetic data and real data; the synthetic data is generated by simulating an oilfield scene and includes images, sensor data and corresponding equipment status labels.

[0010] Furthermore, a preferred method is proposed, wherein the distillation temperature T and the loss weight... These are parameters that are dynamically adjusted based on the correlation of characteristics in multimodal oilfield data. The feature correlation is calculated by measuring the image feature vector. With sensor feature vector Pearson correlation coefficient get; The loss weight The dynamic adjustment formula is: ; The distillation temperature T ranges from 1.0 to 5.0, and the correlation coefficient between the T value and the characteristic is... Negative correlation.

[0011] Furthermore, a preferred method is proposed, wherein the structured pruning of the convolutional layer is performed based on the L1 criterion, calculating the sum of the absolute values ​​of the weights of each channel in the convolutional layer. Remove branches according to the preset pruning ratio r. The smallest channel.

[0012] Furthermore, a preferred method is proposed, wherein the pruning ratio r ranges from 30% to 70%, and in the knowledge distillation optimization training, the loss weight... It is negatively correlated with the pruning ratio r.

[0013] Furthermore, a preferred method is proposed, wherein the dynamic quantization involves converting the 32-bit floating-point weights FP32 and activation values ​​in the model into 8-bit integers INT8, using the following conversion formula: Where S is the quantization scaling factor and Z is the quantization zero. This is the rounding function. This is the original 32-bit floating-point data. This is the compressed 8-bit integer data.

[0014] Furthermore, a preferred method is proposed, wherein the lightweight model conversion includes: operator fusion and redundant node removal of the network structure of the lightweight student model, simultaneous INT8 quantization adaptation of model weights during the conversion process, and operator mapping for edge hardware.

[0015] Based on the same inventive concept, this invention also proposes a multimodal large-scale lightweight distillation optimization system for deployment at the edge of oil fields, the system comprising: The data acquisition unit is used to acquire and process multimodal data from the oilfield, including image data and sensor data. A multimodal model building unit is used to construct a teacher model and a student model based on oilfield multimodal data. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. A distillation unit is used to transfer knowledge from the teacher model to the student model based on knowledge distillation. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The lightweight processing unit is used to sequentially perform convolutional layer structured pruning and dynamic quantization on the student model trained by knowledge distillation to obtain a lightweight model. The output unit is used to convert the lightweight model into a general-format model file suitable for edge inference engines.

[0016] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield as described in any of the preceding claims.

[0017] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield as described in any of the above-mentioned methods.

[0018] Compared with the prior art, the beneficial effects of the present invention are: The method proposed in this invention compresses large multimodal models in the cloud by integrating a collaborative lightweight strategy of knowledge distillation, structured pruning, and dynamic quantization. Experiments show that this method can compress the number of model parameters, computational cost (FLOPs), and storage volume by more than 85%, while maintaining the model's accuracy retention rate of over 98% in oilfield multimodal classification tasks. This effectively solves the problem of balancing high compression rate and high accuracy retention in traditional lightweight techniques.

[0019] By employing a targeted lightweight student model architecture and combinatorial compression techniques, the computational complexity of the model is significantly reduced. After deployment on edge hardware in oilfields, compared to existing general solutions, the average inference time per sample of the lightweight model obtained by this method is significantly shortened, and the inference speed in multimodal tasks can be improved by more than 4 times, perfectly meeting the real-time response requirements of scenarios such as oilfield inspection and equipment monitoring.

[0020] By integrating automated model format conversion capabilities, lightweight models can be converted to common formats such as ONNX, and adaptation processes such as operator mapping and weight solidification are completed. This eliminates compatibility barriers between model formats and various edge inference engines, allowing models to be deployed directly without complex secondary development, improving deployment efficiency by more than 80%.

[0021] By constructing a dual-source data processing mechanism that combines synthetic and real data, it is possible to generate batches of labeled multimodal synthetic data that fits the oilfield scenario, which can then be used in combination with limited real data. This not only expands the training dataset but also covers a wider range of working conditions, thereby improving the adaptability and robustness of the lightweight model in different oilfield edge scenarios. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in this invention. Figure 2 This is a diagram of a multimodal large-scale lightweight distillation optimization system architecture for deployment at the edge of an oilfield, as described in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other, and the described embodiments are only some embodiments of the present invention, not all embodiments.

[0024] Implementation Method 1, see Figure 1 This implementation method addresses the mismatch between existing lightweight model technologies and the actual needs of oilfield edge deployments in key aspects such as architecture adaptation, data provision, and training controllability. It proposes a lightweight distillation optimization method for multimodal large-scale models deployed at oilfield edge locations. The method includes: Acquire and process multimodal data from the oilfield, including image data and sensor data; A teacher model and a student model based on oilfield multimodal data are constructed. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. Knowledge distillation is used to transfer knowledge from the teacher model to the student model. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The student model trained by knowledge distillation is subjected to convolutional layer structured pruning and dynamic quantization in sequence to obtain a lightweight model. The lightweight model is converted into a common format model file suitable for edge inference engines.

[0025] In this embodiment, the oilfield multimodal data is a mixture of synthetic data and real data; the synthetic data is generated by simulating an oilfield scene and includes images, sensor data and corresponding equipment status labels.

[0026] In this embodiment, the distillation temperature T and the loss weight These are parameters that are dynamically adjusted based on the correlation of characteristics in multimodal oilfield data. The feature correlation is calculated by measuring the image feature vector. With sensor feature vector Pearson correlation coefficient get; The loss weight The dynamic adjustment formula is: ; The distillation temperature T ranges from 1.0 to 5.0, and the correlation coefficient between the T value and the characteristic is... Negative correlation.

[0027] In this embodiment, the structured pruning of the convolutional layer is performed based on the L1 criterion, calculating the sum of the absolute values ​​of the weights of each channel in the convolutional layer. Remove branches according to the preset pruning ratio r. The smallest channel.

[0028] In this embodiment, the pruning ratio r ranges from 30% to 70%, and in the knowledge distillation optimization training, the loss weight... It is negatively correlated with the pruning ratio r.

[0029] In this embodiment, the dynamic quantization involves converting the 32-bit floating-point weights FP32 and activation values ​​in the model into 8-bit integers INT8. The conversion formula is as follows: Where S is the quantization scaling factor and Z is the quantization zero. This is the rounding function. This is the original 32-bit floating-point data. This is the compressed 8-bit integer data.

[0030] In this embodiment, the lightweight model conversion includes: performing operator fusion and redundant node removal on the network structure of the lightweight student model, simultaneously completing the INT8 quantization adaptation of the model weights during the conversion process, and performing operator mapping for edge hardware.

[0031] Existing lightweight models are typically simple scaling or cropping of general models. The method proposed in this implementation addresses the unique heterogeneous data streams from images and sensors specific to oilfields. It features a specially designed dual-branch feature extraction teacher model structure, upon which an extremely lightweight student model is constructed while retaining the core multimodal fusion logic. This asymmetric design and knowledge inheritance principle ensure that the student model, even after structural simplification, can still effectively handle the multimodal features specific to oilfields, rather than being a degraded version of general features.

[0032] Existing knowledge distillation techniques mostly employ fixed distillation temperatures and loss weights. This implementation dynamically correlates distillation parameters with the feature correlation coefficients of the oilfield multimodal data itself. When the correlation between image and sensor data features is low, the T-value is automatically increased to smooth the teacher's output, promote cross-modal knowledge transfer, and is adjusted synchronously. To balance losses; conversely, when the correlation is high. This principle of driving the distillation process with data characteristics enables personalized adaptation of distillation strategies to specific oilfield scenario data, making knowledge transfer more accurate and efficient.

[0033] Existing technologies often employ multiple lightweight methods in isolation or simply in series, which can easily lead to error accumulation and performance degradation. This implementation proposes a multi-stage collaborative optimization process: first, a student model with high accuracy potential is trained through feature-adaptive knowledge distillation; then, the model structure is pruned based on the L1 criterion to compress the number of parameters and computational load; subsequently, fine-tuning distillation is performed to restore accuracy; finally, dynamic quantization is used to compress storage volume. This principle clarifies the order and synergistic relationship of each method, achieving a positive superposition of compression effects rather than a simple accumulation of accuracy losses.

[0034] Existing technologies typically rely solely on limited real-world labeled data for model training and lightweighting. This implementation method simulates and generates labeled synthetic multimodal data by pre-setting oilfield scenario parameters (background, sensor range), and then mixes it with real data as needed. This not only supplements the amount of data but also simulates abnormal operating condition samples that may be missing in real-world data through synthetic data, fundamentally solving the problem of poor model generalization ability caused by data scarcity and imbalanced samples.

[0035] Implementation Method 2, see below Figure 2 This embodiment describes a complete implementation process for the multimodal large-scale lightweight distillation optimization method for deployment at the edge of oilfields, as described in Embodiment 1. The process includes: The method described in this embodiment is based on a multimodal large model lightweight distillation optimization system. The system architecture is centered on multimodal fusion and model lightweighting, supported by teacher-student model architecture, knowledge distillation, and pruning quantization. It utilizes edge deployment adaptation and full-process visualization as its operational carriers, constructing a six-layer closed-loop architecture. Data exchange and result feedback are achieved between each layer. The specific architecture layers are as follows: Parameter configuration layer: Enables flexible setting and updating of lightweight training parameters such as distillation temperature, loss weight, pruning ratio, and batch size, and outputs personalized lightweight training parameters for oilfield edge. Multimodal dual-source data processing layer: Enables adaptive generation of oilfield scenario-based synthetic multimodal data and standardized loading of real multimodal data, providing data support for model training and lightweighting; Multimodal model initialization layer: Enables the automated construction of cloud-based multimodal teacher models and edge-based multimodal student models, and outputs a pre-initialized teacher-student model architecture; Model lightweight optimization layer: Integrates three core lightweighting techniques: knowledge distillation, convolutional layer pruning, and dynamic quantization, to achieve the transfer of knowledge from the teacher model to the student model and the dual lightweight optimization of the student model, and outputs a lightweight edge multimodal model; Model Conversion and Evaluation Layer: Enables automatic conversion of lightweight models to the ONNX universal format, while building a performance evaluation system specifically for oilfield edge applications. Simultaneously evaluates model accuracy, inference speed, and model size, and outputs deployable model files and quantitative evaluation reports.

[0036] The multimodal large-model lightweight distillation optimization method described in this embodiment includes: Acquire and process multimodal data from the oilfield, including image data and sensor data; A teacher model and a student model based on oilfield multimodal data are constructed. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. Knowledge distillation is used to transfer knowledge from the teacher model to the student model. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The student model trained by knowledge distillation is subjected to convolutional layer structured pruning and dynamic quantization in sequence to obtain a lightweight model. The lightweight model is converted into a common format model file suitable for edge inference engines.

[0037] This implementation addresses the scarcity of real multimodal data from oilfields by pre-setting a 7:3 mixing ratio of synthetic and real data. Synthetic data comprises 70% to cover sample distributions across different oilfield scenarios and abnormal equipment conditions, resolving the issue of uneven real sample distribution. Real data accounts for 30% to align with the characteristics of real data in the target oilfield, ensuring the model's field adaptability. Simultaneously, it supports adaptive adjustment of the mixing ratio within a range of 6:4 to 8:2 based on the accumulated real data volume in the target oilfield. The process includes: constructing a dual-source supply and mixing mechanism for synthetic and real data; generating multimodal data (images and sensor data) from simulated oilfield inspection scenarios, along with equipment status labels and other annotation information; and supporting standardized format loading and parsing of real data, enabling proportional mixing and seamless switching between the two data sources, providing sufficient and oilfield-specific data support for multimodal model training and lightweight design.

[0038] In this embodiment, the acquisition of synthetic data includes: pre-setting the image background and sensor parameter range for oilfield inspections, including temperature, pressure, and vibration; randomly generating multimodal data fused from images and sensors; automatically labeling equipment status (normal, minor anomaly, severe anomaly); and outputting a fully labeled multimodal synthetic dataset. It supports loading real oilfield inspection images and sensor CSV data in mainstream formats, automatically parsing image features and sensor numerical features, completing data standardization and feature alignment processing, and adapting to the input requirements of multimodal models. This embodiment addresses the scarcity of real multimodal data from oilfields. The synthetic data can batch simulate multimodal data from different oilfield scenarios and equipment states. Real data undergoes automated parsing and feature standardization, improving data processing efficiency by ≥50%.

[0039] In this implementation, the design concept of the teacher model and student model based on oilfield multimodal data is as follows: Based on the characteristics of oilfield multimodal data (images and sensors), a cloud-based teacher model and an edge-based student model are constructed respectively. The teacher model adopts a complex multimodal fusion architecture to achieve deep feature extraction from oilfield multimodal data. The student model simplifies and lightens the architecture of the teacher model, significantly reducing the number of parameters and computational load while retaining the core feature extraction capabilities, making it suitable for low-computing-power hardware at the oilfield edge. Specifically, the teacher model is divided into an image feature extraction branch and a sensor feature extraction branch. The image branch uses a multi-layer convolutional neural network to extract deep visual features, and the sensor branch uses a multi-layer perceptron to extract numerical features. Deep fusion of dual-modal features is achieved through a fusion layer to output high-dimensional multimodal features. The student model simplifies the channels and neurons of the convolutional layers and fully connected layers of the teacher model, reducing the number of network layers and kernels, and simplifying the fusion layer structure. While ensuring that the feature extraction logic is consistent with the teacher model, the architecture is extremely lightweight.

[0040] This implementation addresses the heterogeneity of multimodal data from oilfield images and sensors by designing a dynamic adjustment mechanism for distillation temperature T and loss weight based on modal correlation coefficients. This replaces the traditional fixed-parameter distillation loss design; T and All values ​​are adaptively adjusted based on the feature correlation of multimodal data from the oilfield. When the feature correlation between images and sensor data is low, the T-value is increased to enhance cross-modal knowledge transfer and simultaneously improve... Weighting ensures distillation effectiveness; when feature correlation is high, reduce the T value and decrease... Weighting focuses on optimizing classification accuracy, perfectly adapting to the multimodal data characteristics of different well sites and equipment scenarios in oilfields, improving knowledge transfer effectiveness by ≥60%, and keeping the accuracy loss of student models stably within 5%.

[0041] This implementation proposes a knowledge distillation method that replaces the traditional fixed-parameter distillation loss design. It supports flexible configuration of distillation temperature and loss weights, adapting to the knowledge transfer needs of different oilfield edge scenarios. The composite loss function achieves dual constraints on soft-label knowledge and hard-label accuracy, improving knowledge transfer performance by ≥60% while keeping student model accuracy loss within 5%. The knowledge distillation design proposed in this approach transfers the multimodal deep feature knowledge of the oilfield learned by the cloud-based teacher model to the edge-side student model. By designing a composite loss function with dual-parameter control of temperature and weights, it integrates KL divergence distillation loss (soft-label knowledge transfer) and cross-entropy classification loss (hard-label classification accuracy) to achieve accurate knowledge transfer while suppressing classification errors in the student model. Specifically: Distillation loss (KL divergence): Used to realize the transfer of soft-labeled knowledge from the teacher model, reflecting the similarity between the output distributions of the student and teacher models. The formula is:

[0042] in, For the output of the student model, The output of the teacher model is T, which is the distillation temperature and its value is limited to [1.0, 5.0]. The value of T is negatively correlated with the characteristics of oilfield multimodal data. The lower the correlation, the larger the value of T, the smoother the soft label, which can enhance the transfer effect of cross-modal association knowledge. Classification loss (cross-entropy): Used to ensure the classification accuracy of the student model and constrain its ability to fit hard labels. The formula is:

[0043] Where L represents the true hard label of the data. This is the function for calculating cross-entropy loss; Method for calculating the correlation coefficient of multimodal data features in oilfields: For two types of heterogeneous data, oilfield images and sensor data, the depth visual feature vectors of the image branches are first extracted separately. Numerical feature vectors of the sensor branch The correlation between two types of features is calculated using the Pearson correlation coefficient, as shown in the formula:

[0044] in, The covariance calculation function, ρ is the variance calculation function, and ρ is the correlation coefficient of the features of multimodal data in the oilfield. The value range is ρ∈[0,1]. The larger the value of ρ, the higher the feature correlation between the image and the sensor data.

[0045] The method integrates distillation loss and classification loss, and dynamically adjusts the weighting coefficients based on the correlation of features in multimodal oilfield data.α The formula for achieving a balanced regulation between the two is:

[0046] Among them, weight parameters α The dynamic adjustment formula is: ρ is the Pearson correlation coefficient of the multimodal data characteristics of the above oilfield, corresponding to α The dynamic range of α is [0.2, 0.8]; This is the square coefficient of the distillation temperature, used to compensate for the decrease in value caused by the increase in temperature.

[0047] The layered structured pruning and dynamic quantization processing proposed in this implementation method overcome the limitations of traditional single lightweighting methods, constructing a four-order progressive collaborative lightweighting strategy, and clarifying the execution order and parameter linkage mechanism of each method: The loss weight α in the distillation stage is negatively correlated with the pruning ratio r. When the pruning ratio r increases, the weight of α increases simultaneously, which strengthens the knowledge transfer effect in the distillation stage and offsets the accuracy loss caused by pruning. Distillation temperature T is positively correlated with quantization accuracy. When the number of quantization bits decreases, the T value is increased simultaneously to enhance the smoothness of the soft label and improve the robustness of the quantized model. The pruning ratio r is adaptively adjusted based on the computing power threshold of the oilfield edge hardware, with a range limited to r∈[30%,70%], adapting to edge industrial control computers and AI boards with different computing power. This collaborative strategy achieves triple compression of model parameter quantity, computational quantity, and storage volume, improving lightweight efficiency by ≥40% compared to a single method, while avoiding the invalid calculation of redundant parameters, further improving the inference speed at the edge. Its design concept includes: constructing a four-order collaborative lightweight optimization process of knowledge distillation pre-training, convolutional layer structured pruning, fine-tuning distillation, and dynamic quantization. Knowledge distillation is the core to achieve cross-modal knowledge transfer, convolutional layer pruning is used to compress parameters at the model structure level, and dynamic quantization is used to compress the storage volume of model parameters. Through the sequential execution and parameter linkage of multiple methods, the accuracy loss of oilfield multimodal analysis tasks is strictly controlled while maximizing model compression.

[0048] The lightweight model is obtained as follows: The first stage, knowledge distillation pre-training: Based on the dual-source mixed dataset, the multimodal knowledge transfer from the teacher model to the initial student model is completed through a composite loss function, resulting in a pre-trained student model with complete multimodal feature extraction capabilities, laying the accuracy foundation for subsequent pruning and quantization operations; The second stage, convolutional layer structured pruning: Based on the L1 criterion, channel pruning is performed on the image feature extraction branch convolutional layer of the pre-trained student model. Redundant channels with low contribution to multimodal feature classification are removed according to the preset pruning ratio to achieve lightweight compression of the model structure. The third stage, fine-tuning distillation after pruning: Perform 5 to 10 rounds of lightweight distillation fine-tuning on the pruned student model to repair the accuracy loss caused by the pruning operation and ensure the multimodal feature extraction capability and classification accuracy of the pruned model. The fourth stage, dynamic quantization adaptation, involves dynamically quantizing the weights and activation values ​​of the finely tuned student model, converting the 32-bit floating-point parameters into 8-bit integer parameters, completing a second lightweighting process at the model storage and inference computation levels, and finally obtaining a lightweight model adapted to the edge hardware of the oilfield.

[0049] In this embodiment, the convolutional layer pruning decision is based on the L1 criterion, which calculates the sum of the absolute values ​​of the weights of each channel in the convolutional layer. The formula is as follows:

[0050] in, Let the sum of the absolute values ​​of the weights of the c-th channel be denoted as . The weights of the convolution kernel for the c-th channel are used; remove the weights according to the pruning ratio r. Minimize redundant channels, retain core channels; Dynamic quantization conversion includes converting all 32-bit floating-point (FP32) weights and activation values ​​in the model to 8-bit integers (INT8). The conversion formula is as follows:

[0051] Where S is the quantization scaling factor, and Z is the quantization zero. This is a rounding function; during inference, the INT8 data is converted back to FP32 to ensure calculation accuracy.

[0052] This implementation achieves automatic conversion of lightweight student models to the ONNX universal format, eliminating compatibility barriers between model formats and oilfield edge inference engines. Simultaneously, it constructs a model performance evaluation system specifically for oilfield edge scenarios, evaluating three core indicators: model accuracy, average inference speed per sample, and model size, quantifying the actual effect of model lightweighting and edge adaptation. This implementation replaces the traditional manual model format conversion method, achieving automated format conversion and improving deployment efficiency by ≥80%. It is the first to design a multi-dimensional performance evaluation system specifically for oilfield edge scenarios, rather than a single accuracy evaluation, ensuring the evaluation results better align with the actual application needs of oilfield edge scenarios. Automatic ONNX universal format conversion in this implementation: Based on the network structure and parameters of the lightweight model, it automatically completes the conversion to the ONNX universal format. The conversion process strictly executes three core sub-steps to ensure the accuracy, compatibility, and inference efficiency of the converted model, including: The network structure of the lightweight model is processed by operator fusion, redundant node removal and branch simplification to eliminate useless operators generated during training, simplify the model computation graph and reduce the scheduling overhead during inference. During the format conversion process, the INT8 quantization adaptation of the model weights is completed simultaneously, and the quantization scaling factor S and zero-point Z parameters are fixed to ensure that the converted model can run directly in the edge INT8 inference engine without secondary quantization processing. For the ARM architecture and embedded AI acceleration chips commonly used in oilfield edge computing, standardized mapping and adaptation of operators are completed, replacing incompatible custom operators. This ensures that the converted ONNX model can seamlessly adapt to mainstream edge inference engines such as TensorRT, ONNXRuntime, and NCNN, eliminating deployment format barriers. It also supports dynamic batch size configuration to adapt to different inference batch requirements in oilfield edge computing.

[0053] The method proposed in this embodiment employs multi-dimensional performance evaluation, specifically: The accuracy of the equipment status classification model is evaluated using the accuracy assessment method; the hardware environment at the edge of the oil field is simulated, and the average inference time per sample (ms / sample) of the model is calculated; the file storage volume (MB) after model conversion is statistically analyzed, and the lightweight compression ratio is calculated.

[0054] In this embodiment, an experimental verification was also completed based on a real inspection scenario of an oil production plant in Daqing Oilfield. The experimental data covered the inspection images of the pumping unit (nodding donkey) and the pressure / temperature / vibration sensor data at the wellhead, including three categories of labels: normal operating conditions, minor anomalies, and severe anomalies. The experimental hardware environment was: a cloud training environment with an NVIDIA RTX 3090Ti GPU, and an edge deployment environment with an ARM architecture embedded industrial control computer (4-core CPU, 8G memory, 2TOPS computing power) commonly used in oilfields.

[0055] This experiment tested the classification accuracy of the proposed method and existing general multimodal lightweight distillation schemes for three types of input data: image unimodal, sensor unimodal, and image and sensor multimodal. The model parameter compression ratio and inference speed improvement were also compared simultaneously. The experimental results are shown in the table below:

[0056] The method of this invention significantly outperforms existing general solutions in classification accuracy across three data modalities in oilfields, especially in multimodal data fusion scenarios, where accuracy is improved by 7.56 percentage points. At the same time, it achieves a model parameter compression rate of over 90% and improves edge inference speed by more than 3.7 times, perfectly meeting the real-time requirements of low-computing-power hardware at the edge of oilfields.

[0057] This experiment uses a cloud-trained multimodal teacher model as a benchmark to compare the lightweight performance of the proposed method with existing mainstream edge model compression schemes. Key metrics include the number of model parameters, floating-point computation (FLOPs), the compression ratio of storage volume, and the accuracy retention rate relative to the teacher model. The experimental results are shown in the table below.

[0058] Compared with existing mainstream compression schemes, the method of this invention achieves a higher degree of model compression, with the compression ratio of parameter quantity, computation quantity and storage volume all exceeding 85%, while maintaining the accuracy retention rate at over 98%. It solves the industry pain point that traditional lightweight solutions cannot simultaneously improve compression ratio and maintain accuracy, and fully meets the deployment needs of resource-constrained environments at the edge of oil fields.

[0059] Implementation Method 3: This implementation method proposes a multimodal large-scale lightweight distillation optimization system for deployment at the edge of oilfields. The system includes: The data acquisition unit is used to acquire and process multimodal data from the oilfield, including image data and sensor data. A multimodal model building unit is used to construct a teacher model and a student model based on oilfield multimodal data. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. A distillation unit is used to transfer knowledge from the teacher model to the student model based on knowledge distillation. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The lightweight processing unit is used to sequentially perform convolutional layer structured pruning and dynamic quantization on the student model trained by knowledge distillation to obtain a lightweight model. The output unit is used to convert the lightweight model into a general-format model file suitable for edge inference engines.

[0060] The system proposed in this embodiment builds a visual interactive interface that integrates full-process functions such as parameter configuration, multimodal data management, model building, knowledge distillation, pruning and quantization, format conversion, performance evaluation, and log recording. It realizes one-click operation and real-time display of results for lightweighting of multimodal large models at the oilfield edge, while automatically recording all lightweighting process and result information, and supporting log querying, saving, and traceability.

[0061] It addresses the issues of fragmented processes and high operational barriers in traditional lightweight model implementation, achieving integrated functionality across the entire process. The visual interface design aligns with the operating habits of oilfield maintenance personnel, enabling lightweight model implementation without requiring specialized coding skills, thus reducing operating costs by ≥60%. It also automates the recording of lightweight process logs, resolving the problem of untraceable processes and results.

[0062] The visual interactive interface supports the following functions: Visualized parameter configuration: Supports visual input and modification of core parameters such as distillation temperature, loss weight, pruning ratio, and number of training rounds, with real-time effects; Data visualization management: Displays the sample quantity and feature information of synthetic / real multimodal data, and supports data preview and filtering; Process visualization and monitoring: Real-time display of the training process and progress of knowledge distillation and pruning quantization, outputting key indicators such as loss curves and accuracy curves; Results visualization: The evaluation results of the lightweight model, such as accuracy, inference speed, and model size, are displayed in chart form, intuitively quantifying the optimization effect; Automated logging: Automatically records all information such as operation time, parameter configuration, training process, and evaluation results, and supports querying, clearing, and local saving of logs.

[0063] In practical applications, the system's workflow is as follows: Step S1, System Initialization and Parameter Configuration: Start the visualization system, complete the initialization of equipment and various functional modules, and adapt to the hardware simulation environment at the edge of the oilfield; Configure lightweight core parameters in the visualization interface, including the distillation temperature T and the initial threshold of the loss weight α, which are dynamically adjusted based on the correlation coefficient of oilfield multimodal data features. Initialize the runtime environment for modules such as model building, distillation training, and pruning quantization to prepare for subsequent processes.

[0064] Step S2: Preparation and loading of multimodal dual-source data: Select the multimodal data source through the visual interface: synthetic data or real data; if synthetic data is selected: set the number of samples, the system will generate multimodal synthetic data of images and sensors that fit the oilfield inspection scenario in batches, automatically label the equipment status, and output a synthetic dataset with complete annotations; if real data is selected: select the file path of real images and sensor data, the system will automatically load and parse the data, complete feature standardization and feature alignment, and output a standardized real dataset. Synthetic data and real data were mixed according to a preset 7:3 ratio. The mixed multimodal dataset was then divided into training and testing sets at an 8:2 ratio to provide data support for model distillation training and performance evaluation.

[0065] Step S3: Initialization of the multimodal teacher-student model: Based on the characteristics of multimodal data from the oilfield, the system automatically constructs the architecture of the cloud-based teacher model and the edge-based student model, and completes the initialization of the network layer; The teacher model is pre-trained to enable it to perform deep feature extraction and equipment status classification of multimodal oilfield data; The parameters of the teacher model are fixed to prepare for subsequent knowledge distillation training.

[0066] Step S4, Knowledge Distillation Optimization Training: The training set is input into the initialized teacher-student model, and the soft labels output by the teacher model provide a basis for knowledge transfer to the student model. Based on the designed composite loss function, the total loss value of the student model is calculated, and the parameters of the student model are updated through backpropagation. Iterate through the training process to the preset number of rounds, complete the multimodal knowledge transfer from the teacher model to the student model, and output the pre-trained student model. The accuracy of the student model is verified using the test set. If the accuracy does not reach the preset threshold, the relevant parameters of the distillation temperature T and the loss weight α are readjusted and the model is trained again.

[0067] Step S5: Model pruning and dynamic quantization lightweighting: Convolutional layer structured pruning: For the pre-trained student model, redundant channels of the convolutional layer are removed according to the L1 criterion and a preset pruning ratio r, while retaining the core multimodal feature extraction channels, and the pruned model is output. Post-pruning fine-tuning distillation: Perform 5 to 10 rounds of lightweight distillation fine-tuning on the pruned model, repair the accuracy loss caused by pruning based on the composite loss function, and output the lightweight model with restored accuracy; Dynamic quantization adaptation: Perform dynamic quantization on the fine-tuned model, convert the 32-bit floating-point parameters into 8-bit integer parameters, complete the second lightweight quantization of the model, and output the final lightweight model adapted to the edge. The parameters and structure of the lightweight model are validated to ensure the integrity of the model's network structure and the effectiveness of its multimodal feature extraction capabilities.

[0068] Step S6: Edge Model Conversion and Multi-Dimensional Performance Evaluation The lightweight model is automatically converted into the ONNX universal format to generate model files that can be deployed on the inference engine at the edge of the oil field; Simulate the hardware environment at the edge of an oilfield, use the test set to perform multi-dimensional performance evaluation of the ONNX format model, and simultaneously calculate the model's classification accuracy, average inference time per sample, and model storage volume, and output a quantitative performance evaluation report. If the assessment results do not meet the preset requirements at the edge of the oilfield, return to step S1 to readjust the parameters and execute the lightweight process.

[0069] Implementation Method 4: This implementation method proposes a computer device, including a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in any one of Implementation Methods 1 to 2.

[0070] Implementation Method 5: This implementation method proposes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in any one of Implementation Methods 1 to 2.

[0071] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit its protection scope. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading this disclosure, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims.

Claims

1. A lightweight distillation optimization method for multimodal large-scale models deployed at the edge of oilfields, characterized in that, The method includes: Acquire and process multimodal data from the oilfield, including image data and sensor data; A teacher model and a student model based on oilfield multimodal data are constructed. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. Knowledge distillation is used to transfer knowledge from the teacher model to the student model. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The student model trained by knowledge distillation is subjected to convolutional layer structured pruning and dynamic quantization in sequence to obtain a lightweight model. The lightweight model is converted into a common format model file suitable for edge inference engines.

2. The multimodal large-scale lightweight distillation optimization method for deployment at the edge of oilfields according to claim 1, characterized in that, The oilfield multimodal data is a mixture of synthetic and real data; the synthetic data is generated by simulating oilfield scenarios and includes images, sensor data and corresponding equipment status labels.

3. The multimodal large-scale lightweight distillation optimization method for deployment at the edge of oilfields according to claim 1, characterized in that, The distillation temperature T and loss weight These are parameters that are dynamically adjusted based on the correlation of characteristics in multimodal oilfield data. The feature correlation is calculated by measuring the image feature vector. With sensor feature vector Pearson correlation coefficient get; The loss weight The dynamic adjustment formula is: ; The distillation temperature T ranges from 1.0 to 5.0, and the correlation coefficient between the T value and the characteristic is... Negative correlation.

4. The lightweight distillation optimization method for multimodal large-scale models deployed at the edge of oilfields according to claim 1, characterized in that, The structured pruning of the convolutional layer is performed based on the L1 criterion, calculating the sum of the absolute values ​​of the weights of each channel in the convolutional layer. Remove branches according to the preset pruning ratio r. The smallest channel.

5. The multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in claim 4, is characterized in that... The pruning ratio r ranges from 30% to 70%, and in knowledge distillation optimization training, the loss weight... It is negatively correlated with the pruning ratio r.

6. The multimodal large-scale lightweight distillation optimization method for deployment at the edge of oilfields according to claim 1, characterized in that, The dynamic quantization involves converting the 32-bit floating-point weights FP32 and activation values ​​in the model into 8-bit integers INT8. The conversion formula is as follows: Where S is the quantization scaling factor and Z is the quantization zero. This is the rounding function. This is the original 32-bit floating-point data. This is the compressed 8-bit integer data.

7. The multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in claim 6, is characterized in that... The lightweight model conversion includes: operator fusion and redundant node removal of the network structure of the lightweight student model, simultaneous INT8 quantization adaptation of model weights during the conversion process, and operator mapping for edge hardware.

8. A multimodal large-scale lightweight distillation optimization system for deployment at the edge of oilfields, characterized in that, The system includes: The data acquisition unit is used to acquire and process multimodal data from the oilfield, including image data and sensor data. A multimodal model building unit is used to construct a teacher model and a student model based on oilfield multimodal data. The teacher model includes an image feature extraction branch and a sensor feature extraction branch, and features are fused through a fusion layer. The student model is a lightweight architecture obtained by simplifying the channels, neurons, and network layers of the teacher model. A distillation unit is used to transfer knowledge from the teacher model to the student model based on knowledge distillation. The knowledge distillation is trained using a composite loss function, which includes distillation loss and classification loss. The lightweight processing unit is used to sequentially perform convolutional layer structured pruning and dynamic quantization on the student model trained by knowledge distillation to obtain a lightweight model. The output unit is used to convert the lightweight model into a general-format model file suitable for edge inference engines.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield, as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of a multimodal large-scale lightweight distillation optimization method for deployment at the edge of an oilfield as described in any one of claims 1-6.

Images