A pumping unit operation state time sequence-visual fusion light-weighted inference system and method for handheld terminals

Abstract

The application discloses a pumping unit operation state time sequence-visual fusion lightweight reasoning system and method for a handheld terminal, and belongs to the technical field of oilfield fault diagnosis. The prior art has low composite fault recognition accuracy, high multi-modal model computing power requirement, and cannot be used in real-time operation of a handheld terminal. Conventional lightweight means will also destroy the modal feature matching relationship. The application is optimized from three dimensions of lightweight network, adaptive cross-modal fusion and joint lightweight optimization. A data acquisition module synchronously acquires pumping unit time sequence sensing data and appearance images; a double-branch feature extraction module respectively extracts features and outputs single-modal features with consistent dimensions; a cross-modal attention fusion module adaptively allocates channel weights, splices and transforms to generate fusion features; a fault classification module outputs diagnosis results based on the fusion features; a joint lightweight module compresses the network and guarantees feature alignment; and a terminal deployment module deploys the lightweight model to a handheld terminal, so that end-side real-time fault reasoning is realized.

Description

Technical Field

[0001] This invention belongs to the technical field of oilfield equipment fault diagnosis, multimodal feature fusion and edge model lightweighting, specifically involving a lightweight reasoning method and system for time-vision fusion of pumping unit operating status for handheld terminals. Background Technology

[0002] Pumping units are core production equipment in oilfield crude oil extraction, and their operating status directly determines the oilfield's production efficiency and safety level. Currently, pumping unit fault diagnosis mainly falls into two technical categories: one based on time-series data from sensors such as vibration and current to diagnose internal operating conditions, and the other based on visual images to identify external defects. With the advancement of intelligent inspection in oilfields, diagnostic solutions relying on a single data source can no longer meet the needs of refined on-site operation and maintenance. Existing technologies have the following core shortcomings in practical applications: 1. Insufficient accuracy of single-sensor diagnostics and lack of ability to identify complex faults. Current pumping unit fault diagnosis relies heavily on single vibration or current sensors, which can only capture numerical characteristics of internal equipment operation and cannot identify complex faults such as gear wear accompanied by belt slippage, or bearing failure accompanied by seal leakage. The fault diagnosis accuracy of single-sensor solutions is generally below 78%, with a high rate of missed detections, making it difficult to support comprehensive equipment condition assessment. Currently, most well sites in Daqing Oilfield still rely primarily on manual, experience-based inspections, resulting in delayed fault detection and a high rate of misjudgment.

[0003] 2. The temporal-visual fusion model architecture is complex, making deployment on edge portable terminals difficult. Existing multimodal fusion diagnostic models mostly employ a stacked architecture of deep convolutional networks and long short-term memory networks, resulting in a large number of parameters and high computational complexity. They are typically deployed on cloud servers or fixed industrial control cabinets, making them unsuitable for the low-computing-power, low-power hardware environment of handheld inspection terminals in oilfields. Testing shows that general multimodal fusion models achieve inference speeds below 4fps on mainstream edge handheld terminals, failing to meet the diagnostic requirements of real-time inspections.

[0004] 3. Existing lightweight methods have poor adaptability and are prone to disrupting cross-modal fusion structures. Traditional lightweight model methods are mostly designed for single-modal vision models. When directly applied to temporal-visual dual-branch fusion models, they easily disrupt the feature alignment and fusion association between the two modalities, resulting in a significant decrease in fault diagnosis accuracy. In particular, the accuracy of early minor faults and composite faults is reduced by more than 11%, making it impossible to balance lightweight compression with the maintenance of diagnostic accuracy.

[0005] 4. Lack of end-to-end integrated solutions for handheld inspection scenarios. Most existing diagnostic systems are designed for fixed-point monitoring and do not consider the operation process, hardware limitations, and field environment characteristics of handheld inspections. They cannot achieve integrated portable operation of "collection-analysis-diagnosis-recording" and are difficult to adapt to the actual operation and maintenance needs of Daqing Oilfield, which has dispersed well sites, long inspection routes, and imperfect network coverage.

[0006] To address the shortcomings of the existing technologies, this invention proposes a lightweight inference algorithm for the time-vision fusion of pumping unit operation status for handheld terminals. It breaks through from three dimensions: lightweight network architecture, adaptive cross-modal fusion, and joint lightweight optimization, solving the industry pain points of low accuracy in composite fault identification and difficulty in deploying multimodal models in real time on handheld terminals. Summary of the Invention

[0007] To address the core shortcomings of existing oil pumping unit fault diagnosis technologies, this invention aims to achieve the following objectives: 1. Construct a lightweight dual-branch feature extraction architecture, and optimize the temporal and visual feature extraction networks through depthwise separable convolutions, thereby significantly reducing model complexity while ensuring feature representation capabilities; 2. Design a channel attention-driven cross-modal adaptive fusion mechanism to achieve complementary fusion of temporal operation features and visual appearance features, thereby improving the diagnostic accuracy of complex faults; 3. A dual-branch joint lightweight strategy is proposed, which achieves high-precision real-time inference in the low computing power environment of handheld terminals through the synergistic optimization of structured pruning and hybrid precision quantization. 4. Develop an end-to-end pumping unit fault diagnosis solution adapted to domestically produced handheld terminals to meet the engineering application needs of portable field inspections in Daqing Oilfield.

[0008] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a lightweight inference system for the time-vision fusion of oil pumping unit operation status for handheld terminals, the inference system comprising: The data acquisition module is used to simultaneously acquire multi-dimensional time-series sensor data and appearance image data of the oil pumping unit; A lightweight dual-branch feature extraction module is used to extract features from the collected multi-dimensional time-series sensor data and appearance image data, and output two types of single-modal features with uniform dimensions. The cross-modal attention adaptive fusion module is used to adaptively allocate channel-dimensional weights to the two types of single-modal features respectively, and then achieve cross-modal deep fusion through feature concatenation and nonlinear transformation. The fault classification output module receives multimodal fusion features and outputs the fault diagnosis results of the pumping unit. The dual-branch joint lightweight optimization module is used to compress and optimize the overall network composed of the aforementioned feature extraction and fusion, while maintaining the alignment and matching relationship between the two types of features during the compression process. The terminal inference deployment module is used to deploy the lightweight optimized network to handheld terminals to achieve real-time fault inference on the terminal side.

[0009] Furthermore, the data acquisition module of the present invention includes a vibration sensor, a current sensor and a visible light camera, used to simultaneously acquire vibration timing data, current timing data and appearance images of the oil pumping unit.

[0010] Furthermore, the data acquisition module of the present invention also includes a preprocessing unit, which is used to perform noise reduction, normalization and length alignment preprocessing on the vibration time series data and current time series data, and to perform size normalization and pixel value standardization preprocessing on the appearance image.

[0011] Furthermore, the lightweight dual-branch feature extraction module of the present invention includes: The temporal feature extraction branch adopts an improved temporal convolutional network, which consists of four stacked causal dilated convolutional layers. Each convolutional layer is followed by a ReLU activation function and a one-dimensional max pooling layer. The standard convolution is replaced with a depthwise separable convolution. Finally, a one-dimensional temporal feature vector is output through global average pooling and a fully connected layer. The visual feature extraction branch employs a lightweight visual network with depthwise separable convolutions as the basic computational unit. A lightweight channel attention module is integrated into each bottleneck structure. Channel weights are generated through global average pooling and two fully connected layers. After multi-level downsampling, a visual feature vector with the same dimension as the temporal feature vector is output through global pooling and fully connected layers.

[0012] Furthermore, the temporal feature extraction branch and the visual feature extraction branch of the present invention are used to extract features from the multi-dimensional temporal sensing data and the appearance image data, respectively, and both the temporal feature extraction branch and the visual feature extraction branch adopt a lightweight network structure.

[0013] Furthermore, the cross-modal attention adaptive fusion module of this invention employs a channel attention-driven adaptive fusion mechanism, adaptively allocating channel-dimensional weights to the features of the two modalities respectively, specifically as follows: Single-modal channel weight calculation: Global average pooling is performed on temporal features and visual features respectively to compress the features into a global description vector of the channel dimension. Then, through a shared two-layer fully connected network and a Sigmoid activation function, temporal channel weight vectors and visual channel weight vectors are generated respectively. Feature weighting: The generated channel weights are multiplied channel by channel with the original features of the corresponding modality to complete the adaptive weighting of features; Fusion output: The weighted temporal features and visual features are concatenated along the channel dimension, and then nonlinearly fused and dimensionally transformed through a fully connected layer to output the final multimodal fused features.

[0014] Furthermore, the dual-branch joint lightweight optimization module of this invention employs a joint lightweight strategy combining structured pruning and mixed precision quantization to perform end-to-end compression on the overall network constructed by the aforementioned feature extraction and fusion, specifically including: Structured pruning: The importance of each channel in the two branch convolutional layers is evaluated using the L1 regularization criterion. The L1 norm of each channel weight is calculated as the importance score. Redundant channels with importance scores below the threshold are removed according to the preset pruning ratio, while core channels that contribute highly to fault diagnosis are retained. Hybrid precision quantization: A hierarchical hybrid precision quantization strategy is adopted, using INT8 integer quantization for dual-branch convolutional and pooling layers, and FP16 half-precision floating-point quantization for cross-modal attention layers and classification fully connected layers. Fine-tuning and repair involves fine-tuning the pruned and quantized model to repair the loss of accuracy until the diagnostic accuracy meets the preset requirements, at which point the model is solidified.

[0015] Furthermore, the terminal inference deployment module of the present invention is specifically used for: optimizing the computation graph and adapting operators to the lightweight model, including operator fusion, redundant node removal and hardware instruction set adaptation, converting the model into an inference format adapted to the edge NPU, and deploying it to the handheld terminal.

[0016] Furthermore, the fault classification output module of the present invention includes a fully connected classifier and a Softmax function, used to receive the multimodal fusion features, calculate the probability value of each fault type, and output the diagnostic results and corresponding confidence levels of the pumping unit's operating status.

[0017] Based on the same inventive concept, this invention also provides a lightweight inference method for time-vision fusion of oil pumping unit operation status for handheld terminals, the method comprising the following steps: S1: Data acquisition and preprocessing steps: Vibration time series data, current time series data and appearance image of the oil pumping unit are acquired simultaneously through the vibration sensor, current sensor and visible light camera integrated in the handheld terminal; the time series data is preprocessed by denoising, normalization and length alignment, and the image is preprocessed by size normalization and pixel value standardization to obtain standardized dual-modal input data. S2: Dual-branch feature extraction step: Input the preprocessed vibration and current time series data into the improved temporal convolutional network temporal feature branch in the lightweight dual-branch feature extraction module to extract multi-scale temporal correlation features and output a one-dimensional temporal feature vector; Input the preprocessed appearance image into the lightweight visual feature branch in the lightweight dual-branch feature extraction module to extract appearance semantic features and output a visual feature vector with the same dimension as the temporal feature vector; S3: Cross-modal adaptive fusion step: Input the temporal feature vector and the visual feature vector into the cross-modal attention adaptive fusion module, perform global average pooling in the channel dimension respectively, generate their respective channel weight vectors through a shared fully connected network and Sigmoid activation function, multiply the channel weight vectors with the original features of the corresponding modality channel by channel to complete adaptive weighting, and then concatenate the weighted temporal features and visual features in the channel dimension and perform nonlinear transformation through a fully connected layer to output multimodal fusion features; S4: Fault classification and diagnosis step, input the multimodal fusion features into the fully connected classifier in the fault classification output module, calculate the probability value of each fault type through the Softmax function, and output the diagnosis result of the pumping unit's operating status and the corresponding confidence level; S5: Model joint lightweighting step. Based on the trained dual-branch fusion model, structured pruning and mixed precision quantization are performed through the dual-branch joint lightweighting optimization module. Structured pruning evaluates the importance of each channel of the two-branch convolutional layer based on the L1 regularization criterion and removes redundant channels according to a preset ratio. Mixed precision quantization uses INT8 integer quantization for computationally intensive layers and FP16 half-precision floating-point quantization for precision-sensitive layers. After fine-tuning and repairing the precision loss, a lightweight diagnostic model is obtained. S6: Terminal deployment inference step. The lightweight diagnostic model is converted into an inference format adapted to the edge NPU through the terminal inference deployment module and deployed to the handheld terminal. Data is collected in real time during on-site inspection and end-to-end inference is performed to output the diagnostic results of the pumping unit's operating status.

[0018] The beneficial effects of this invention are as follows: 1. Significantly improved accuracy in composite fault diagnosis. Through adaptive fusion of time-series operating characteristics and visual appearance characteristics, complementary analysis of internal operating status and external appearance defects of equipment is achieved, increasing the accuracy of composite fault diagnosis by 22% to 99.2%. It can accurately identify 10 types of single and composite faults, such as gear wear, belt slippage, and bearing failure, effectively reducing the missed detection rate.

[0019] 2. The model achieves significant lightweighting, and the inference speed meets real-time requirements. Through a lightweight architecture design with depthwise separable convolutions and joint optimization with pruning quantization, the number of model parameters is reduced by more than 50%, and the inference speed is improved by 2.3 times. On the handheld terminal RK3588NPU, the inference speed reaches more than 9fps, and the status analysis time for a single pumping unit is less than 0.2 seconds, which fully meets the needs of real-time field inspection.

[0020] 3. Lightweighting with controllable accuracy loss. The dual-branch joint lightweighting strategy is designed to adapt to the dual-modal fusion structure. The pruning process maintains the dimensionality matching of the fusion layer, and the mixed precision quantization preserves the accuracy of the attention layer. The diagnostic accuracy loss of the lightweight model is controlled within 1%, achieving a balance between compression ratio and accuracy.

[0021] 4. Compatible with domestically produced handheld terminals, ensuring strong engineering feasibility. The solution is fully compatible with the business scenarios of handheld inspections in Daqing Oilfield, supports the deployment of domestically produced edge AI chips, does not rely on cloud networks, and can complete the entire process of diagnosis offline in the field well site, significantly improving the intelligence level and operational efficiency of pumping unit inspections.

[0022] This invention is applicable to field inspection scenarios in oilfields. Based on handheld terminals, it enables real-time acquisition of the operating status of pumping units, multimodal fusion analysis, and accurate fault diagnosis. It is especially compatible with low-computing-power hardware environments using domestically produced edge AI chips, and can support portable and intelligent inspection operations of pumping units in Daqing Oilfield. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is the overall architecture diagram of the lightweight inference system for oil pumping unit operation status fusion based on time-vision for handheld terminals according to the present invention. Figure 2 This is a flowchart of the lightweight inference method for oil pumping unit operation status fusion based on time-vision for handheld terminals according to the present invention. Figure 3 This is a schematic diagram of the temporal feature extraction branch of the present invention; Figure 4 This is a schematic diagram of the cross-modal attention adaptive fusion module of the present invention; Figure 5 This is a biaxial bar chart comparing the performance of the dual-branch lightweight architecture of this invention. Figure 6This is a bar chart comparing the performance of the fusion methods of the present invention, wherein... Figure 6 (a) in the figure is a bar chart comparing the performance of different fusion methods. Figure 6 (b) in the figure is a stacked histogram of typical fault mode weight distributions; Figure 7 This is a line graph comparing the accuracy loss before and after pruning in this invention. Figure 8 This is a dual-axis comparison chart of the accuracy and computing power utilization rate of the quantification scheme of this invention; Figure 9 This is a comparison chart of the collaborative performance thresholds of all modules in this invention. Detailed Implementation

[0025] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0026] Example 1, Combination Figures 1 to 4 This embodiment addresses the core shortcomings of existing pumping unit fault diagnosis technologies by proposing a lightweight inference system and method for handheld terminals that integrates time-series and visual data to determine the operating status of pumping units. The system aims to achieve the following objectives: 1. Construct a lightweight dual-branch feature extraction architecture, and optimize the temporal and visual feature extraction networks through depthwise separable convolutions, thereby significantly reducing model complexity while ensuring feature representation capabilities; 2. Design a channel attention-driven cross-modal adaptive fusion mechanism to achieve complementary fusion of temporal operation features and visual appearance features, thereby improving the diagnostic accuracy of complex faults; 3. A dual-branch joint lightweight strategy is proposed, which achieves high-precision real-time inference in the low computing power environment of handheld terminals through the synergistic optimization of structured pruning and hybrid precision quantization. 4. Develop an end-to-end pumping unit fault diagnosis solution adapted to domestically produced handheld terminals to meet the engineering application needs of portable field inspections in Daqing Oilfield.

[0027] The lightweight inference system includes: The data acquisition module is used to simultaneously acquire multi-dimensional time-series sensor data and appearance image data of the oil pumping unit; A lightweight dual-branch feature extraction module is used to extract features from the collected multi-dimensional time-series sensor data and appearance image data, and output two types of single-modal features with uniform dimensions. The cross-modal attention adaptive fusion module is used to adaptively allocate channel-dimensional weights to the two types of single-modal features respectively, and then achieve cross-modal deep fusion through feature concatenation and nonlinear transformation. The fault classification output module receives multimodal fusion features and outputs the fault diagnosis results of the pumping unit. The dual-branch joint lightweight optimization module is used to compress and optimize the overall network composed of the aforementioned feature extraction and fusion, while maintaining the alignment and matching relationship between the two types of features during the compression process. The terminal inference deployment module is used to deploy the lightweight optimized network to handheld terminals to achieve real-time fault inference on the terminal side.

[0028] Furthermore, in a specific embodiment, the lightweight inference system described in this embodiment will be specifically described; like Figure 1 As shown, this invention is based on a closed-loop process of "dual-source data preprocessing - lightweight dual-branch feature extraction - cross-modal attention adaptive fusion - fault classification output - dual-branch joint lightweighting - terminal deployment inference". Through lightweight network architecture design, adaptive cross-modal fusion mechanism and joint lightweighting optimization, it achieves high-precision real-time diagnosis of the pumping unit's operating status in a low-computing-power hardware environment of a handheld terminal. The core technical modules and implementation steps are as follows: 1. Core Functional Modules (1) Lightweight dual-branch feature extraction module This module includes a temporal feature extraction branch and a visual feature extraction branch, which extract features from the multi-dimensional temporal sensing data and appearance image data of the oil pumping unit, respectively. Both branches adopt a lightweight network structure and output single-modal features with uniform dimensions, laying the foundation for subsequent cross-modal fusion.

[0029] 1.1 Temporal Feature Branch of Improved Temporal Convolutional Network like Figure 3 As shown, in view of the long-period correlation characteristics of time series data such as pumping unit vibration and current, an improved temporal convolutional network is used as the backbone for temporal feature extraction. The traditional causal convolution is replaced with a depthwise separable convolutional structure, which significantly reduces the amount of computation and parameters while retaining the advantages of causality and the large receptive field of dilated convolution.

[0030] Network structure: A 4-layer stacked causal dilated convolutional structure is adopted, with a kernel size of 3 and dilation coefficients of 1, 2, 4 and 8 respectively, to expand the receptive field layer by layer and cover the multi-timescale features of the pumping unit operation; after each convolutional layer, a ReLU activation function and a one-dimensional max pooling layer are connected to enhance key temporal features and perform dimensionality reduction.

[0031] Depthwise separable convolution optimization: Traditional standard convolution is split into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs spatial convolution independently on each input channel, while pointwise convolution combines features along the channel dimension using 1×1 convolutions. Let the number of input feature channels be... The number of output channels is The kernel size is The timing length is Then the computational cost of standard convolution is The computational cost of depthwise separable convolution is The number of parameters and computational load can be reduced by more than 70%.

[0032] Output processing: After four layers of temporal convolution, a one-dimensional temporal feature vector is obtained through global average pooling, and then mapped to a unified feature dimension through a fully connected layer, outputting temporal features consistent with the visual branch dimension.

[0033] 1.2 Lightweight Visual Feature Extraction Branch To address the feature extraction requirements of oil pumping unit appearance images, a lightweight visual network is used as the backbone. An efficient feature extraction structure is constructed through depthwise separable convolution and channel attention mechanism. The width coefficient is set to 0.5 to reduce the number of network channels, making it suitable for the computing power level of handheld terminals.

[0034] Basic Unit: Depth-separable convolution is used as the basic computational unit, including 3×3 depthwise convolution and 1×1 pointwise convolution, with batch normalization and h-swish activation function, adapted to the high-contrast characteristics of oilfield images.

[0035] Channel attention enhancement: A lightweight channel attention module is integrated into each bottleneck structure. Channel weights are generated through global average pooling and two fully connected layers to strengthen the feature weights of key areas such as pumping unit belts, gearboxes, and balance blocks, and suppress interference from background irrelevant information.

[0036] Feature output: After multiple levels of downsampling, the network extracts global pooling features, maps them through a fully connected layer to a unified feature dimension consistent with the temporal branch, and outputs a visual feature vector.

[0037] (2) Cross-modal attention adaptive fusion module like Figure 4 As shown, an adaptive fusion mechanism driven by channel attention is designed to address the heterogeneity of temporal and visual features. The mechanism adaptively allocates channel-dimensional weights to the features of the two modalities, highlighting the feature channels that contribute highly to fault diagnosis. Then, cross-modal deep fusion is achieved through feature concatenation and nonlinear transformation.

[0038] Single-modal channel weight calculation: for temporal features With visual features Global average pooling is performed separately to compress the features into channel-dimensional global description vectors. These vectors are then passed through a shared two-layer fully connected network and a sigmoid activation function to generate temporal channel weight vectors. With visual channel weight vector The weight values ​​range from [0,1]. The weight calculation formula is:

[0039] in, For global average pooling, This is the Sigmoid activation function.

[0040] Feature weighting: The generated channel weights are multiplied with the original features of the corresponding modes channel by channel to complete the adaptive weighting of features, strengthen the key fault feature channels, and suppress redundant interference channels.

[0041] Fusion output: The weighted temporal features and visual features are concatenated along the channel dimension, and then nonlinearly fused and dimensionally transformed through a fully connected layer to output the final multimodal fusion features, which are then input into the downstream fault classifier.

[0042] (3) Dual-branch joint lightweight optimization module To address the deployment requirements of handheld terminals, a joint lightweighting strategy combining structured pruning and hybrid precision quantization is adopted to compress the dual-branch fusion model end-to-end, while ensuring the integrity of the cross-modal fusion structure and diagnostic accuracy.

[0043] Structured pruning: The importance of each channel in the two branch convolutional layers is evaluated using the L1 regularization criterion. The L1 norm of each channel weight is calculated as an importance score. Redundant channels with importance scores below a preset pruning ratio are removed, while core channels that contribute significantly to fault diagnosis are retained. The formula for calculating the channel importance score is as follows:

[0044] in, For the first The first channel One weight parameter, Score the importance of this channel.

[0045] In this invention, the overall pruning ratio is set to 50%. During the pruning process, the channel dimension matching of the cross-modal fusion layer is maintained to avoid disrupting the feature alignment relationship of the two branches.

[0046] Hybrid precision quantization: A layered hybrid precision quantization strategy is adopted. INT8 integer quantization is used for computationally intensive layers such as dual-branch convolutional layers and pooling layers to compress storage volume and computational cost; FP16 half-precision floating-point quantization is used for precision-sensitive layers such as cross-modal attention layers and fully connected classification layers to ensure the accuracy of fused feature representation and classification accuracy. The quantization mapping formula is:

[0047] in, These are the original floating-point parameters. This is the quantization scaling factor. To quantize the zero point, This is a rounding function. During inference, it converts the integer value to a floating-point value through dequantization to complete the calculation, balancing inference speed and accuracy.

[0048] (4) Terminal inference deployment module To address the hardware characteristics of domestically produced edge AI chips, the lightweight model undergoes computational graph optimization and operator adaptation. The model is then converted into an inference format compatible with edge NPUs, enabling efficient inference on handheld terminals. Optimizations include operator fusion, redundant node removal, and hardware instruction set adaptation, fully utilizing NPU computing resources to improve inference efficiency.

[0049] Furthermore, in one specific embodiment, this embodiment provides a lightweight inference method for the timing-visual fusion of oil pumping unit operating status for handheld terminals, such as... Figure 2 As shown, the method includes the following steps: S1: Data Acquisition and Preprocessing Vibration time-series data, current time-series data, and appearance images of the oil pumping unit are simultaneously collected using vibration sensors, current sensors, and visible light cameras integrated into the handheld terminal. The time-series data is preprocessed by denoising, normalization, and length alignment, while the images are preprocessed by size normalization and pixel value standardization to obtain standardized dual-modal input data.

[0050] S2: Bi-branch feature extraction The preprocessed vibration and current time series data are input into the improved TCN time series feature branch to extract multi-scale time series correlation features; the preprocessed appearance image is input into the lightweight visual feature branch to extract appearance semantic features; the two branches output single-modal feature vectors with consistent dimensions.

[0051] S3: Cross-modal adaptive fusion Temporal and visual features are input into the cross-modal attention fusion module. The channel weights of the two modalities are calculated separately through the channel attention mechanism to complete the adaptive weighting of features. Then, the features are concatenated and nonlinearly transformed to output multimodal fusion features.

[0052] S4: Fault Classification and Diagnosis The multimodal fusion features are input into a fully connected classifier, and the probability values ​​of each fault type are calculated through the Softmax function. The diagnostic results of the pumping unit's operating status and the corresponding confidence scores are then output.

[0053] S5: Joint Lightweighting of Models Based on the trained dual-branch fusion model, structured pruning is performed to remove redundant channels, followed by mixed precision quantization compression to obtain a lightweight diagnostic model. The lightweight model is then fine-tuned to repair the accuracy loss, and the model is solidified after verifying that the accuracy meets the requirements.

[0054] S6: Handheld Terminal Deployment Inference The lightweight model is converted into an inference format adapted to the edge NPU and deployed to a handheld terminal. During on-site inspections of the terminal, data is collected in real time and end-to-end inference is performed to output the diagnostic results of the pumping unit's operating status.

[0055] Implementation 2: This embodiment combines a specific application scenario, taking the handheld inspection scenario of an oil pumping unit in an oil production plant of Daqing Oilfield as the implementation environment, to specifically describe the present invention; The hardware platform uses a domestically produced handheld terminal with an integrated RK3588 chip. The terminal is equipped with a vibration sensor (sampling rate 1kHz), a current sensor (sampling rate 100Hz), and a 1080P visible light camera. The operating system is OpenEuler Industrial Edition, and the NPU computing power is 6TOPSINT8.

[0056] This embodiment diagnoses 10 operating states of the pumping unit, including: normal operation, belt slippage, belt breakage, gear wear, bearing failure, motor overheating, balance weight misalignment, seal leakage, base looseness, and gearbox oil shortage, which includes multiple compound fault conditions.

[0057] Step 1: Dataset Construction and Preprocessing This embodiment constructs a multimodal dataset of pumping units containing 12,000 samples. Each sample includes synchronously collected vibration time-series data, current time-series data, and an external image of the pumping unit. All samples are labeled with fault type and severity level by oilfield operation and maintenance experts. Among them, there are 3,000 normal samples, 6,000 single fault samples, and 3,000 compound fault samples, which are divided into training and test sets in an 8:2 ratio.

[0058] The data preprocessing process is as follows: Time series data preprocessing: Wavelet thresholding is used to remove environmental interference from vibration and current data, and then Z-score normalization is used to eliminate dimensional differences. The formula is:

[0059] in, The mean of the data. Standard deviation To prevent division by zero for minimum values, all time series data are aligned to 1024 time steps. Short series are padded with linear interpolation, and long series are truncated at the end.

[0060] Image data preprocessing: The appearance image is uniformly scaled to a resolution of 224×224 pixels, converted to RGB three-channel format, and the pixel values ​​are normalized and mapped to the [0,1] interval.

[0061] Step 2: Construction of a lightweight dual-branch feature extraction network (1) Improved TCN timing feature branch An improved temporal convolutional network with 4 stacked layers is constructed. The input is temporal data with 2 channels (vibration and current) and a length of 1024.

[0062] First layer: dilation coefficient 1, kernel size 3, number of output channels 32, depthwise separable convolution, followed by ReLU activation and max pooling with a pooling kernel of 2, output time length 512; Second layer: dilation coefficient 2, kernel size 3, number of output channels 64, depthwise separable convolution, followed by ReLU activation and max pooling with a pooling kernel of 2, output time length 256; The third layer has an inflation factor of 4, a kernel size of 3, 128 output channels, a depthwise separable convolution, followed by ReLU activation and max pooling with a kernel size of 2, and an output time length of 128. Fourth layer: dilation factor 8, kernel size 3, output channels 256, depthwise separable convolution, followed by ReLU activation; After global average pooling, the vector is fed into a fully connected layer and mapped to a 256-dimensional temporal feature vector.

[0063] This branch uses depthwise separable convolution to replace the traditional standard convolution. While maintaining the same receptive field and feature extraction capability, the number of parameters is reduced from 126,000 to 32,000, and the computational cost is reduced by 74.6%.

[0064] (2) Lightweight visual feature branch A lightweight visual network is constructed based on depthwise separable convolution, with a width factor of 0.5 and an input of a 224×224×3 RGB image.

[0065] The network consists of five bottleneck stages, each composed of multiple depthwise separable convolutional units, which are downsampled by a factor of 2 at each stage, with the number of channels being 16, 24, 40, 80, and 160 respectively. Each bottleneck unit is connected to a channel attention module to adaptively weight the channel features.

[0066] Finally, after global average pooling, it is connected to the fully connected layer and mapped to a 256-dimensional visual feature vector, which is consistent with the temporal branch dimension.

[0067] Step 3: Construction of Cross-Modal Attention Adaptive Fusion Module The cross-modal fusion module takes 256-dimensional temporal features and 256-dimensional visual features as inputs, and the fusion process is as follows: 1. Perform global average pooling on the feature vectors of the two modalities respectively, and input them into a shared two-layer fully connected network. The first layer has 32 neurons and the second layer has 256 neurons. After Sigmoid activation, 256-dimensional temporal channel weights and visual channel weights are generated respectively.

[0068] 2. Multiply the weight vector element-wise with the original features of the corresponding modality to complete the adaptive weighting of the features and strengthen the feature dimensions that contribute highly to fault diagnosis.

[0069] 3. The weighted temporal features and visual features are concatenated along the channel dimension to obtain 512-dimensional concatenated features. Then, a 256-dimensional fully connected layer is used for nonlinear fusion to output 256-dimensional multimodal fusion features.

[0070] 4. This fusion mechanism can automatically learn the contribution weights of two modes under different fault types. For example, for belt slippage faults, it automatically increases the weight of the belt region in visual features; for bearing faults, it automatically increases the weight of vibration time sequence features, achieving adaptive complementarity of cross-modal features.

[0071] Step 4: Fault Classifier and Model Training After fusing features, a fully connected classifier is connected, with an output dimension of 10, corresponding to 10 types of pumping unit operating states. The probability value of each type is output through the Softmax function.

[0072] The model training uses cross-entropy loss as the supervised loss function, and AdamW is used as the optimizer. The initial learning rate is set to 0.001, and cosine annealing is used to decay the learning rate. The training consists of 50 epochs with a batch size of 32. An early stopping strategy is used during training: training is terminated when the accuracy on the validation set no longer improves for 5 consecutive epochs, and the optimal model weights are saved.

[0073] Step 5: Joint Lightweight Optimization of Two Branches (1) Structured pruning The L1 criterion is used to evaluate the channel importance of all convolutional layers in the dual-branch system. The L1 norm of each channel weight is calculated as the importance score, and redundant channels with the lowest scores are removed according to a 50% pruning ratio. During the pruning process, the output feature dimensions of the temporal branch and the visual branch are kept consistent to ensure that the input dimensions of the cross-modal fusion layer match and avoid destroying the fusion structure.

[0074] After pruning, the model was fine-tuned for 10 rounds to repair the accuracy loss caused by pruning. The fine-tuning learning rate was set to 0.0001.

[0075] (2) Hybrid precision quantization The pruned model underwent hierarchical mixed-precision quantization: all convolutional and pooling layers used INT8 quantization, while fully connected layers across modal attention layers and classifier fully connected layers used FP16 quantization. Parameter calibration was performed using a calibration dataset to determine the quantization scaling factor for each layer. With zero point To ensure the accuracy of quantification.

[0076] The storage volume of the quantized model is compressed by 75%, and the amount of computation is greatly reduced. At the same time, the precision-sensitive layer retains half precision, controlling the overall precision loss.

[0077] Step 6: Handheld Terminal Deployment and Inference The lightweight model was converted to an inference format adapted for the RK3588 NPU, and operator fusion and computation graph optimization were performed. Continuous operators such as convolution, batch normalization, and activation were fused to reduce NPU instruction scheduling overhead. After deployment, the inference flow of the handheld terminal is as follows: 1. Upon arrival at the pumping unit site, inspection personnel simultaneously collect vibration and current data and visual images via terminal sensors; 2. The terminal automatically completes data preprocessing and inputs it into the lightweight diagnostic model; 3. The model extracts features through a two-branch process, performs cross-modal fusion, and classifies and infers the results of the pumping unit's operating status diagnosis, fault type, and confidence level. 4. The diagnostic results are displayed visually on the terminal interface. Abnormal conditions will automatically trigger audible and visual alarms, and inspection records will be saved to local storage.

[0078] Furthermore, in this embodiment, the performance of the solution described in this invention is verified on a test set and compared with existing mainstream solutions. The results are shown in Table 1 below: Table 1

[0079] The comparison results show that the composite fault diagnosis accuracy of the present invention is significantly higher than that of the single data source solution. At the same time, compared with the traditional non-lightweight fusion model, the inference speed is increased by 2.3 times to 9fps, and the analysis time of a single device is less than 0.2 seconds, which fully meets the needs of real-time inspection of handheld terminals. The number of model parameters is compressed by 54.7%, which is adapted to the storage and computing power limitations of edge terminals.

[0080] Meanwhile, field tests showed that this solution can operate continuously and stably for more than 8 hours in the field well site environment of Daqing Oilfield, and the terminal endurance meets the needs of all-day inspection. The diagnostic results are 98.7% consistent with the expert's manual judgment, which has extremely high engineering application value.

[0081] Example 3, Combination Figures 5 to 9 This embodiment demonstrates the irreplaceability of the five core innovative modules of the present invention one by one through the control variable method, proving that there is a strong synergistic effect among the modules. It shows that the comprehensive performance of "99.2% composite fault accuracy + 9fps real-time inference" cannot be achieved by simply splicing together existing technologies. Therefore, the solution does not belong to conventional technology improvement.

[0082] Test hardware: A domestically produced handheld terminal with an integrated RK3588 chip (NPU computing power 6 TOPS INT8, memory 8GB, operating system OpenEuler 22.03 LTS Industrial Edition). Test dataset: Multimodal test dataset of pumping unit in an oil production plant of Daqing Oilfield (2400 test samples, covering 10 operating states, including 600 samples of composite faults and 480 samples of early minor faults. All samples have been labeled and confirmed by oilfield operation and maintenance experts). Unified benchmark description: All experiments adopt a unified training strategy (AdamW optimizer, cross-entropy loss function, 50 training rounds, early stopping mechanism), unified data preprocessing process, and unified testing environment. Only experimental variables are changed to ensure that the results are comparable and reproducible.

[0083] Experiment 1: Ablation Experiment of Dual-Branch Lightweight Architecture 1.1 Experimental Objective The proposed architecture of "4-layer dilatation depth separable TCN temporal branch + channel attention lightweight visual branch" demonstrates a synergistic improvement in diagnostic accuracy, model size, and inference speed compared to existing single-modal solutions and traditional simple dual-branch splicing solutions, proving that this architecture is not a simple replacement for conventional networks.

[0084] 1.2 Experimental Design Three control groups and one experimental group were set up. All models used the same training and testing conditions, with only the feature extraction branch structure being changed: Control group 2-1: Single temporal diagnostic model only (traditional standard convolutional single-layer TCN, without depth separability or dilated structure) Control group 2-2: Single visual diagnostic model only (original MobileNetV2, without channel attention enhancement) Control group 1: Traditional single-layer TCN + ordinary MobileNet visual branch (existing technology is simple to stitch together, without depth separation optimization or channel attention) Experimental group: This invention features a 4-layer dilatation depth-separable TCN temporal branch + channelized attention lightweight visual dual branch. 1.3 Experimental Procedure 1. Train four sets of models on a unified training set, using the same optimizer and loss function, until convergence; 2. Convert all four models to an inference format compatible with the RK3588 NPU and perform the same operator fusion optimization. 3. Conduct inference tests on a unified test set. Each experiment is repeated 10 times and the average value is taken. Record four indicators: composite fault diagnosis accuracy, number of model parameters, single-frame inference time, and inference frame rate.

[0085] 1.4 Experimental Data Recording As shown in Table 2 (Statistical Table of Ablation Experiment Results for Dual-Branch Lightweight Architecture); Table 2

[0086] 1.5 Data Analysis and Visualization like Figure 5 As shown, Figure 5 Middle horizontal axis: 4 experimental schemes; Left vertical axis: composite fault diagnosis accuracy (%); Right vertical axis: inference frame rate (fps). from Figure 5 As can be seen, the simple splicing dual-branch scheme only improves accuracy by 6.2 to 10.7 percentage points compared to the single-modal scheme, and the frame rate drops to 5.3 fps, which cannot meet the ≥8 fps requirement for real-time inspection of handheld terminals. In contrast, the architecture of this invention reduces the number of parameters by 15.2% and increases the inference frame rate by 69.8% compared to the simple splicing scheme. At the same time, the accuracy of composite faults is improved by 7.6 percentage points, achieving a synergistic effect of "double improvement in accuracy and speed", rather than the conventional trade-off of one increasing at the expense of the other.

[0087] 1.6 Experimental Conclusions The existing simple bi-branch splicing technology can only achieve a shallow combination of single-modal features, which cannot balance accuracy and terminal real-time performance, and is insufficient in the ability to identify complex faults. This invention achieves synergistic optimization of temporal branches and enhanced channel attention of visual branches through depthwise separable convolution. It is not a simple application of conventional lightweight components, but a customized design for the long-cycle characteristics of oil pumping units and the characteristics of key visual regions. It realizes the synergistic optimization of accuracy and computing power and does not belong to the conventional improvement methods in this field.

[0088] Experiment 2: Cross-modal channel attention adaptive fusion ablation experiment 2.1 Experimental Objective The "dual-channel independent attention weighting + nonlinear fusion" mechanism proposed in this invention demonstrates a significant improvement in the identification of complex faults and early minor faults compared to existing technologies such as simple splicing fusion, fixed weight fusion, and decision-level fusion. This proves that the fusion mechanism has unexpected technical effects.

[0089] 2.2 Experimental Design Three control groups and one experimental group were set up. All models used the same bi-branch feature extraction backbone, only the cross-modal fusion method was changed: Control group 1: Simple feature concatenation and fusion (only channel-level concatenation is performed before being fed into the classifier, without adaptive allocation of channel weights, corresponding to the basic fusion logic in comparison file 2) Control group 2-1: Global fixed weight fusion (temporal features: visual features = 1:1 fixed weighting and then concatenation, without adaptive adjustment) Control group 2-2: Late-stage fusion at the decision level (the two branches output classification results separately, and the fusion is performed using a voting method, without feature-level fusion) Experimental group: This invention uses dual-channel independent attention weighting + fully connected nonlinear fusion. 2.3 Experimental Procedure The weights of the dual-branch feature extraction network are fixed, and only the fusion module is replaced. The parameters of the fusion layer are then fine-tuned based on a unified training set. Test the overall diagnostic performance on the test set, focusing on the accuracy of composite faults and the early minor fault identification rate. For four typical fault types (belt slippage, bearing failure, gear wear, and seal leakage), the weight allocation ratio of the two modes was statistically analyzed to verify the adaptive characteristics.

[0090] 2.4 Experimental Data Recording As shown in Table 3 (Statistical Table of Ablation Experiment Results for Cross-Modal Fusion Methods); Table 3

[0091] As shown in Table 4 (Statistical Table of Modal Weight Allocation under Typical Fault Types); Table 4

[0092] Figure 6 The chart shown is a performance comparison bar chart of the fusion methods, in which... Figure 6 (a) is a bar chart comparing the performance of fusion methods. The horizontal axis represents the four fusion schemes, and the vertical axis represents the accuracy (%). It also shows the comparison of two groups of bars: composite faults and early minor faults. from Figure 6 As can be seen in (a), the composite fault accuracy of the three conventional fusion schemes is less than 92%, and the early minor fault identification rate is less than 72%, which cannot meet the needs of refined inspection in oil fields. Compared with the optimal conventional scheme, the adaptive fusion scheme of this invention improves the composite fault accuracy by 7.5 percentage points and the early minor fault identification rate by 20.9 percentage points, which is far greater than the conventional expectation.

[0093] Figure 6 (b) is a stacked bar chart of typical fault modal weight distribution. The horizontal axis represents 4 types of typical faults, and the vertical axis represents the weight percentage (%). The stacked bar chart displays the temporal and visual weights. It can be seen that the adaptive fusion feature of this invention is that it automatically increases the visual modal weight (≥70%) for appearance-related faults (belt slippage, seal leakage) and automatically increases the temporal modal weight (≥65%) for internal operation-related faults (bearing failure, gear wear). Therefore, this invention can automatically adjust the modal contribution according to the fault type, automatically increase the visual weight for appearance-related faults, and automatically increase the temporal weight for internal operation-related faults, thus achieving cross-modal complementary enhancement.

[0094] 2.6 Experimental Conclusions The conventional fusion methods of existing technologies cannot adaptively allocate modal contributions for different fault types and have insufficient ability to capture fine-grained features of early minor faults. The channel attention adaptive fusion mechanism of the present invention achieves dynamic complementarity of "internal operating features + external appearance features", which significantly improves the identification of early minor faults and complex faults. This is an unexpected technical effect that cannot be easily derived by those skilled in the art through existing fusion methods.

[0095] Experiment 3: Two-branch adaptive structured pruning ablation experiment 3.1 Experimental Objective The verification shows that when general unconstrained pruning is directly applied to a bimodal fusion model, it will destroy the cross-modal feature alignment structure and cause the network to fail. It proves that the constraint pruning proposed in this invention, which is "L1 channel scoring + forced bi-branch dimension matching", is a specific optimization for bi-branch fusion architecture and is not a simple transfer of conventional pruning techniques.

[0096] 3.2 Experimental Design Three groups were set up: a baseline group, a control group, and an experimental group. All groups adopted a uniform pruning ratio of 50%, with only the pruning constraint rules being changed. Baseline: Unpruned, complete two-branch fusion model Control group: General unconstrained pruning (independent channel pruning is performed on temporal and visual branches, without restrictions on output dimension matching, corresponding to the single-model pruning approach in comparison file 3) Experimental group: This invention uses constraint pruning (L1 channel importance scoring, and the pruning process forces the output feature dimensions of both branches to be consistent). 3.3 Experimental Procedure Based on the fully trained baseline model, two pruning strategies were executed, with the pruning ratio uniformly set to 50%. After pruning, the same fine-tuning strategy (10 rounds of fine-tuning, learning rate 0.0001) was used to restore accuracy; Test the fusion effectiveness, diagnostic accuracy loss, and model compression rate of the pruned model.

[0097] 3.4 Experimental Data Recording As shown in Table 5 (Statistical Table of Ablation Experiment Results by Structured Pruning Method); Table 5

[0098] Figure 7 The graph shown is a line graph comparing the accuracy loss before and after pruning according to the present invention. The horizontal axis represents the pruning ratio (0%~60%), and the vertical axis represents the accuracy rate of composite fault diagnosis (%). The two curves correspond to general pruning and constrained pruning, respectively. It can be seen that the accuracy of general unconstrained pruning drops sharply after the pruning ratio exceeds 35%, and the fusion structure is basically ineffective at 50%. In contrast, the accuracy of the constrained pruning of the present invention decreases gradually and linearly within the pruning ratio range of 0~50%, with an accuracy loss of only 0.7% at 50%, while ensuring the integrity of the fusion structure.

[0099] 3.6 Experimental Conclusions Existing single-model pruning techniques cannot be directly transferred to dual-modal fusion models. Unconstrained pruning will disrupt the feature dimension alignment of the two branches, causing the cross-modal fusion mechanism to fail and the diagnostic accuracy to plummet. This invention presents a dimension constraint pruning strategy specifically designed for dual-branch fusion architectures, which resolves the technical contradiction between "compressing the model and maintaining the fusion structure." It is an exclusive innovation for the specific architecture of this solution and does not belong to conventional technical means in this field.

[0100] Experiment 4: Layered Mixing Precision Quantitative Ablation Experiment 4.1 Experimental Objective Verification shows that global uniform INT8 full quantization will severely reduce the feature representation accuracy of cross-modal fusion; it proves that the hierarchical quantization strategy of "INT8 for convolution / pooling layers and FP16 for attention / classification layers" proposed in this invention is a customized optimization for dual-modal fusion models and a simple application of unconventional quantization schemes.

[0101] 4.2 Experimental Design Three groups were set up: a baseline group, a control group, and an experimental group. Quantization was performed uniformly based on the pruned model, with only the quantization rules changed. Benchmark group: Full FP32 floating-point model (unquantized) Control group: Globally uniform INT8 full quantization (all layers use INT8 integer quantization, corresponding to the general quantization scheme in comparison file 3) Experimental group: This invention uses layered hybrid precision quantization (convolutional / pooling layers INT8, cross-modal attention layer, and classification fully connected layer FP16). 4.3 Experimental Procedure Based on the same pruned model, two quantization strategies were executed, and the same calibration dataset was used to calibrate the quantization parameters. The quantized model was deployed to the RK3588 NPU to test inference performance. The overall accuracy, early minor fault identification rate, NPU computing power utilization, and model storage volume compression rate were statistically analyzed.

[0102] 4.4 Experimental Data Recording Table 6 (Statistical Table of Ablation Experiment Results by Quantitative Method) shows the results. Table 6

[0103] 4.5 Data Analysis and Visualization Figure 8 The chart shown is a dual-axis comparison of the accuracy and computing power utilization of the quantification scheme. The left axis (blue): composite fault accuracy (%), range: 80% ~ 100%; the right axis (orange): NPU computing power utilization (%), range: 0% ~ 100%. The bar chart is a side-by-side comparison; within the same group, the left bar represents accuracy (blue), and the right bar represents computing power utilization (orange). It can be seen that... While global INT8 full quantization achieves the highest NPU utilization (78%), the accuracy of composite fault detection (FP32) drops by 4.5 percentage points (99.2% → 94.7%). The attention layer is sensitive to quantization errors, and global INT8 amplifies these errors, causing the early minor fault detection rate to drop from 92.4% to 76.3%, a decrease of 16.1 percentage points. Layered mixed-precision quantization achieves an NPU utilization of 72%, only 6 percentage points lower than global INT8, but its composite fault accuracy reaches 98.6%, a loss of only 0.6 percentage points relative to FP32 (controlled within 1%). Layered quantization also retains good early fault detection capability: 89.8%, significantly better than global INT8 (76.3%).

[0104] 4.6 Experimental Conclusions The general full quantization scheme does not take into account the high sensitivity of cross-modal attention layers to feature accuracy. Direct application will lead to a significant decrease in the ability to express fused features, especially the loss of fine-grained diagnostic capabilities for early faults. The hierarchical quantization strategy of this invention is custom-designed for the characteristics of bimodal models, namely "computationally intensive convolutional layers and precision-sensitive attention layers". It is not a conventional choice for general quantization techniques, and its effect of balancing speed and accuracy is an unexpected technical improvement.

[0105] Experiment 5: Full-Module Overall Collaborative Ablation Experiment 5.1 Experimental Objective By sequentially removing individual innovation modules and replacing them with corresponding existing technologies, it was verified that there is a strong synergistic dependency among the four innovation modules. Only by combining all the modules can the overall effect of "high precision + real-time inference" be achieved. Simply piecing together existing technologies cannot achieve the same performance.

[0106] 5.2 Experimental Design Based on the complete solution of this invention, individual core innovative modules were removed one by one and replaced with conventional solutions from existing technologies. A total of 5 sets of experiments were set up: Complete Solution Group: A combination of all innovative modules of this invention Remove the lightweight TCN group: Replace the depthwise separable dilated TCN with a traditional standard convolutional TCN, and retain the remaining modules. Remove attention fusion group: Replace adaptive attention fusion with simple feature concatenation, and retain the remaining modules. Remove constrained pruning groups: Replace constrained pruning with general unconstrained pruning, retaining the remaining modules. Remove hierarchical quantization group: Replace hierarchical quantization with global INT8 full quantization, retaining the remaining modules. 5.3 Experimental Procedure Five sets of models were constructed respectively, and training, pruning, quantization and terminal deployment were completed according to the corresponding schemes; Test composite fault accuracy, inference frame rate, and model parameter count on a unified test set; Using the "real-time inspection requirements of handheld terminals" as the judgment criteria (accuracy ≥ 95%, inference frame rate ≥ 8fps), we verified whether each group of solutions met the engineering requirements.

[0107] 5.4 Experimental Data Recording As shown in Table 7 (Statistical Table of Experimental Results of Full-Module Collaborative Ablation); Table 7

[0108] 5.5 Data Analysis and Visualization Figure 9 The chart shown is a comparison of the collaborative performance thresholds of the entire module of this invention. The horizontal axis represents 5 experimental schemes; the left vertical axis represents the composite fault accuracy (%), marked with a 95% pass threshold; the right vertical axis represents the inference frame rate (fps), marked with an 8fps pass threshold. It can be seen that only the complete scheme falls within the pass area of ​​"accuracy ≥ 95% and frame rate ≥ 8fps". Removing any of the innovative modules resulted in significant shortcomings such as "inadequate accuracy" or "inadequate speed," failing to meet the engineering application requirements of handheld terminals. The four modules are not simply an additive effect, but rather form a closed-loop synergistic effect: "lightweight architecture ensures basic speed → adaptive fusion ensures core accuracy → constraint pruning and compression do not disrupt fusion → layered quantitative deployment does not lose accuracy." None of them can be omitted.

[0109] 5.6 Experimental Conclusions The four core innovative modules of this invention have strong technical correlation and synergistic effect. Using any one module alone or splicing together existing technologies in a scattered manner cannot simultaneously achieve high accuracy and real-time terminal inference. The simple combination of existing technologies (the multi-source fusion scheme of prior art 2 + the lightweight method of prior art 3) will cause the fusion structure to be destroyed by pruning and the fusion accuracy to be lost by quantization, resulting in a sharp drop in overall performance and failing to achieve the technical effect of this application. This solution achieves a comprehensive performance of "99.2% composite fault accuracy + 9fps real-time inference", which is the result of the synergistic effect of each module and has outstanding substantive features and significant progress.

[0110] In summary, this invention, through a complete set of ablation experiments, demonstrates that the four core innovative modules of this invention have a strong synergistic dependency. Existing technologies, with their fragmented splicing, suffer from problems such as fusion structure failure, a sharp drop in accuracy, and inference lag, failing to simultaneously achieve a composite fault accuracy of 99.2% and real-time terminal inference at 9fps. This solution resolves the core technical contradiction between "multimodal fusion accuracy" and "lightweight edge terminal," possessing outstanding substantive characteristics.

[0111] In response to the objection that "the parameters are common and conventional selections": Through parameter screening experiments, it has been proven that all the key values ​​in this application are the optimal solutions for the specific scenario of "oil pumping unit operating characteristics + RK3588 handheld terminal computing power", with clear critical effects and adaptability, and do not belong to the conventional selections in this field.

[0112] Significant technological advancements: Compared to existing mainstream fusion solutions, this solution improves the accuracy of composite fault diagnosis by 6.7 percentage points and the inference speed by 2.3 times. At the same time, it keeps the accuracy loss after weight reduction to within 1%. It can directly support the engineering implementation of handheld inspection in the field of Daqing Oilfield, and has significant practical value and technological advancements.

[0113] The above description of the technical solution provided by the present invention through several specific embodiments is intended to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, reasonable combinations of implementation methods and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight inference system for the timing-visual fusion of oil pumping unit operation status for handheld terminals, characterized in that, The system includes: The data acquisition module is used to simultaneously acquire multi-dimensional time-series sensor data and appearance image data of the oil pumping unit; A lightweight dual-branch feature extraction module is used to extract features from the collected multi-dimensional time-series sensor data and appearance image data, and output two types of single-modal features with uniform dimensions. The cross-modal attention adaptive fusion module is used to adaptively allocate channel-dimensional weights to the two types of single-modal features respectively, and then achieve cross-modal deep fusion through feature concatenation and nonlinear transformation. The fault classification output module receives multimodal fusion features and outputs the fault diagnosis results of the pumping unit. The dual-branch joint lightweight optimization module is used to compress and optimize the overall network composed of the aforementioned feature extraction and fusion, while maintaining the alignment and matching relationship between the two types of features during the compression process. The terminal inference deployment module is used to deploy the lightweight optimized network to handheld terminals to achieve real-time fault inference on the terminal side.

2. The system according to claim 1, characterized in that, The data acquisition module includes a vibration sensor, a current sensor, and a visible light camera, used to simultaneously acquire vibration time-series data, current time-series data, and appearance images of the oil pumping unit.

3. The system according to claim 2, characterized in that, The data acquisition module also includes a preprocessing unit, which is used to perform noise reduction, normalization and length alignment preprocessing on the vibration time series data and current time series data, and to perform size normalization and pixel value standardization preprocessing on the appearance image.

4. The system according to claim 1, characterized in that, The lightweight dual-branch feature extraction module includes: The temporal feature extraction branch adopts an improved temporal convolutional network, which consists of four stacked causal dilated convolutional layers. Each convolutional layer is followed by a ReLU activation function and a one-dimensional max pooling layer. The standard convolution is replaced with a depthwise separable convolution. Finally, a one-dimensional temporal feature vector is output through global average pooling and a fully connected layer. The visual feature extraction branch employs a lightweight visual network with depthwise separable convolutions as the basic computational unit. A lightweight channel attention module is integrated into each bottleneck structure. Channel weights are generated through global average pooling and two fully connected layers. After multi-level downsampling, a visual feature vector with the same dimension as the temporal feature vector is output through global pooling and fully connected layers.

5. The system according to claim 4, characterized in that, The temporal feature extraction branch and the visual feature extraction branch are used to extract features from the multi-dimensional temporal sensing data and the appearance image data, respectively.

6. The system according to claim 1, characterized in that, The cross-modal attention adaptive fusion module employs a channel attention-driven adaptive fusion mechanism, adaptively allocating channel-dimensional weights to the features of the two modalities, specifically as follows: Single-modal channel weight calculation: Global average pooling is performed on temporal features and visual features respectively to compress the features into a global description vector of the channel dimension. Then, through a shared two-layer fully connected network and a Sigmoid activation function, temporal channel weight vectors and visual channel weight vectors are generated respectively. Feature weighting: The generated channel weights are multiplied channel by channel with the original features of the corresponding modality to complete the adaptive weighting of features; Fusion output: The weighted temporal features and visual features are concatenated along the channel dimension, and then nonlinearly fused and dimensionally transformed through a fully connected layer to output the final multimodal fused features.

7. The system according to claim 1, characterized in that, The dual-branch joint lightweight optimization module employs a joint lightweight strategy combining structured pruning and mixed-precision quantization to perform end-to-end compression on the overall network constructed from the aforementioned feature extraction and fusion. Specifically, it includes: Structured pruning: The importance of each channel in the two branch convolutional layers is evaluated using the L1 regularization criterion. The L1 norm of each channel weight is calculated as the importance score. Redundant channels with importance scores below the threshold are removed according to the preset pruning ratio, while core channels that contribute highly to fault diagnosis are retained. Hybrid precision quantization: A hierarchical hybrid precision quantization strategy is adopted, using INT8 integer quantization for dual-branch convolutional and pooling layers, and FP16 half-precision floating-point quantization for cross-modal attention layers and classification fully connected layers. Fine-tuning and repair involves fine-tuning the pruned and quantized model to repair the loss of accuracy until the diagnostic accuracy meets the preset requirements, at which point the model is solidified.

8. The system according to claim 1, characterized in that, The terminal inference deployment module is specifically used to: optimize the computation graph and adapt operators for the lightweight model, including operator fusion, redundant node removal and hardware instruction set adaptation, convert the model into an inference format adapted to the edge NPU, and deploy it to the handheld terminal.

9. The system according to claim 1, characterized in that, The fault classification output module includes a fully connected classifier and a Softmax function, which are used to receive the multimodal fusion features, calculate the probability value of each fault type, and output the diagnostic results and corresponding confidence scores of the pumping unit's operating status.

10. A lightweight inference method for the timing-visual fusion of oil pumping unit operating status based on the system described in any one of claims 1-9, characterized in that, The method includes the following steps: S1: Data acquisition and preprocessing steps: Vibration time series data, current time series data and appearance images of the oil pumping unit are collected simultaneously through vibration sensors, current sensors and visible light cameras integrated into the handheld terminal. The time-series data is preprocessed by denoising, normalization, and length alignment, and the image is preprocessed by size normalization and pixel value standardization to obtain standardized bimodal input data. S2: Dual-branch feature extraction step, input the preprocessed vibration and current time series data into the improved time series convolutional network time series feature branch in the lightweight dual-branch feature extraction module, extract multi-scale time series correlation features, and output a one-dimensional time series feature vector; The preprocessed appearance image is input into the lightweight visual feature branch of the lightweight dual-branch feature extraction module to extract appearance semantic features and output a visual feature vector with the same dimension as the temporal feature vector. S3: Cross-modal adaptive fusion step: Input the temporal feature vector and the visual feature vector into the cross-modal attention adaptive fusion module, perform global average pooling in the channel dimension respectively, generate their respective channel weight vectors through a shared fully connected network and Sigmoid activation function, multiply the channel weight vectors with the original features of the corresponding modality channel by channel to complete adaptive weighting, and then concatenate the weighted temporal features and visual features in the channel dimension and perform nonlinear transformation through a fully connected layer to output multimodal fusion features; S4: Fault classification and diagnosis step, input the multimodal fusion features into the fully connected classifier in the fault classification output module, calculate the probability value of each fault type through the Softmax function, and output the diagnosis result of the pumping unit's operating status and the corresponding confidence level; S5: Model joint lightweighting step. Based on the trained dual-branch fusion model, structured pruning and mixed precision quantization are performed through the dual-branch joint lightweighting optimization module. Structured pruning evaluates the importance of each channel of the two-branch convolutional layer based on the L1 regularization criterion and removes redundant channels according to a preset ratio. Mixed precision quantization uses INT8 integer quantization for computationally intensive layers and FP16 half-precision floating-point quantization for precision-sensitive layers. After fine-tuning and repairing the precision loss, a lightweight diagnostic model is obtained. S6: Terminal deployment inference step. The lightweight diagnostic model is converted into an inference format adapted to the edge NPU through the terminal inference deployment module and deployed to the handheld terminal. Data is collected in real time during on-site inspection and end-to-end inference is performed to output the diagnostic results of the pumping unit's operating status.

Images