Lightweight semantic pixel joint coding method and system based on vcm and computer readable medium

Abstract

The application provides a VCM-based lightweight semantic pixel joint coding method and system and a computer readable medium, the method first pre-processes power inspection video frames collected; then, a retinal double-guide attention enhancement mechanism is constructed, visual saliency features and power equipment exclusive ROI mask features are fused, power core target features are adaptively enhanced, and scene background redundant information is suppressed; then, a semantic-pixel double-branch parallel coding architecture is built, fine semantic features are purified and refined through an embedded semantic flow representation module, and high-definition pixel detail representation features are reserved through a pixel flow representation module, so that two types of independent code streams suitable for different power businesses are generated; finally, a perception-reconstruction-code rate ternary collaborative constraint loss function is constructed, end-to-end collaborative iterative optimization is performed on global network parameters of the coding framework, and multi-index balanced optimization of coding performance is realized. The application optimizes the balance among perception accuracy, fidelity and code rate overhead, and improves the scene adaptability of power video coding.

Description

Technical Field

[0001] This invention belongs to the field of video coding and computer vision intelligent perception technology, specifically relating to a lightweight semantic pixel joint coding method and system based on VCM. Background Technology

[0002] With the rapid popularization of smart grid panoramic inspection technology, the power inspection video data generated by equipment such as drone inspections, robot inspections, and online monitoring terminals is experiencing explosive growth. Video encoding and compression, as a core pre-processing technology for power video transmission, storage, and intelligent analysis, directly determines the accuracy of power grid operation and maintenance perception, the fidelity of video traceability, and bandwidth transmission efficiency. Existing mainstream video encoding technologies are mostly general-purpose encoding frameworks designed for civilian video scenarios. They have significant technical shortcomings in the complex scenarios specific to the power industry and cannot adapt to the diversified business needs of power grid inspections, which require "high-precision perception, high-fidelity traceability, and low-bandwidth transmission." These limitations are specifically reflected in the following three aspects: 1) The general coding model has poor adaptability to power scenarios and insufficient ability to extract core target features. Existing video coding methods mostly employ fixed-structure feature extraction networks and uniform coding strategies, failing to optimize for features specific to power grid inspection scenarios. Power grid inspection footage exhibits significant scene-specific characteristics, including slender conductors, point insulators, irregular line obstructions, large areas of background vegetation, and redundant sky regions, among other differentiated features. General-purpose models cannot effectively distinguish between core power equipment targets and redundant background information, resulting in issues such as the easy loss of key defect features, incomplete extraction of features from irregularly shaped targets, and over-coding of invalid background features. This leads to a significant decrease in the accuracy of power defect identification and equipment status perception after coding, making it difficult to support intelligent operation and maintenance perception tasks for the power grid.

[0003] 2) The coding mode is too simple and cannot meet the dual needs of intelligent sensing and manual traceability. Current power video coding technologies generally employ a single-stream coding architecture, enabling only single-dimensional video compression output. If the focus is on bitrate compression efficiency, significant amounts of equipment detail, texture, and edge features are lost, failing to meet the high-fidelity requirements of manual fault tracing and verification. Conversely, if the focus is on video reconstruction quality, the coding bitrate increases dramatically, consuming substantial transmission bandwidth and resulting in high latency and severe stuttering in low-bandwidth scenarios in remote inspection areas. This single coding mode is ill-suited for the dual-application scenarios of "machine intelligent sensing and precise manual tracing" in power services, exhibiting extremely poor business adaptability.

[0004] 3) The optimization constraint dimension is too singular, and the multi-core performance indicators cannot be balanced to the optimal level.

[0005] Existing video coding models often focus on pixel reconstruction errors or single image quality metrics in their loss optimization functions, achieving only a single dimension of video quality optimization. However, power grid inspection video coding involves three mutually constraining core metrics: perception accuracy, reconstruction fidelity, and bitrate overhead. Traditional single-constraint methods cannot achieve coordinated and balanced optimization of multiple metrics, easily leading to technical contradictions such as "meeting image quality standards but missing defects, or accurate perception but excessively high bitrate," making it difficult to meet the needs of large-scale, routine, and high-precision smart grid inspection operations. Summary of the Invention

[0006] To address the technical pain points of poor adaptability to various scenarios, limited coding modes, and inability to achieve balanced optimization of multiple performance indicators in power video coding technologies, this invention proposes a lightweight semantic-pixel joint coding method and system based on VCM (Video Coding for Machines). This method strengthens the core target features of power systems through a retina-like dual-guided attention enhancement mechanism, builds a semantic-pixel dual-branch parallel coding architecture to adapt to dual business requirements, and constructs a ternary collaborative constraint loss function to achieve balanced optimization of multiple indicators, effectively solving many shortcomings of existing technologies.

[0007] The technical solution of this invention: A lightweight semantic pixel joint encoding method based on VCM, including an offline model training and convergence process and an online encoding process for inspection videos: (1) Model offline training convergence process: Construct a fully labeled power inspection dataset and preprocess it, build a VCM coding network including a retina-like dual-guided attention enhancement module, a semantic stream representation module, and a pixel stream representation module; construct a perception-reconstruction-bitrate ternary collaborative constraint loss function, input standardized power video frames into the network to perform feature enhancement and dual-branch coding to obtain dual-stream bitstreams, calculate the total loss of the ternary collaborative constraint, and iteratively update all network parameters through backpropagation until the model's perception accuracy, reconstruction fidelity, bitrate overhead, and other performance indicators converge in a balanced manner, and save the optimal coding model; (2) Online encoding process for inspection videos: Step 1: Real-time acquisition of power inspection video stream, reading time-series video images frame by frame, completing size standardization, pixel normalization and noise filtering, removing invalid abnormal frames such as black screen, overexposure, and severe blur, and generating standardized video frames after the above preprocessing. Step 2: Based on the retina-like dual-guided attention enhancement mechanism, the standardized video frames are processed. Multi-scale basic features of the standardized video frames are extracted through a lightweight backbone network. The image visual saliency features and power equipment-specific ROI mask features are fused to generate an adaptive attention weight matrix. This completes the enhancement of core power target features and the suppression of background redundancy features, resulting in attention-enhanced features. Step 3: The attention-enhancing features are processed synchronously using a semantic-pixel dual-branch parallel VCM coding framework. The attention-enhancing features are input into the semantic stream representation module and the pixel stream representation module respectively. The semantic representation features are purified by the multi-scale context enhancement module built into the semantic stream representation module. After dimensionality compression and adaptive quantization, a lightweight semantic code stream is generated. The attention-enhancing features are reconstructed by the pixel stream representation module. After lightweight encoding and quantization, a high-fidelity pixel code stream is generated. Step 4: Adaptive transmission bitstream. Switch between single-stream or dual-stream encrypted transmission based on the on-site network bandwidth. In low-bandwidth environments, only the semantic bitstream is transmitted. When bandwidth is sufficient, both the semantic bitstream and the pixel bitstream are transmitted simultaneously.

[0008] Preferably, the ternary collaborative constraint loss function adopts a multi-objective joint optimization loss model, breaking the limitations of traditional coding's single image quality and single precision. It specifically balances the three mutually restrictive core indicators of power intelligent sensing accuracy, video reconstruction fidelity, and coding transmission bitrate overhead. The overall total loss calculation formula is as follows: In the formula, This represents the total end-to-end iterative loss of the model. The loss in power sensing tasks is used to constrain the accuracy of power equipment defect detection and condition identification. This is the pixel reconstruction loss, used to constrain the pixel detail and structural restoration quality of video frames; This is a dual-stream collaborative bit rate constraint loss, used to manage the bandwidth overhead of semantic stream and pixel stream encoding transmission; This is a dynamically adjustable adaptive weight balancing coefficient. Based on the core requirements of the power industry—"perception first, image quality second, and extreme speed control"—the optimal value is selected. Prioritize ensuring the accuracy of power defect detection and equipment identification, while also considering video reconstruction effects and low bitrate compression performance.

[0009] Preferably, the The calculation formula integrates the dual-task constraints of defect detection and equipment status classification: In the formula, A dedicated combined loss for wire defects is employed, fusing CIoU bounding box regression loss and cross-entropy classification loss to constrain the accuracy of localization and recognition of subtle defects. A cross-entropy loss function is adopted for insulator state-specific classification loss to constrain classification accuracy and adapt to the differentiated sensing tasks of power equipment. , for , The weighting coefficients are preferred. Prioritize optimizing defect detection accuracy.

[0010] Preferably, the The loss branch, which balances pixel accuracy and structural integrity, is calculated using the following formula: In the formula, The mean squared error loss is used to constrain the pixel grayscale deviation between the original video frame and the reconstructed video frame. It is a structural similarity index used to constrain the consistency of the edges, textures, and contours of power equipment; , for , The weighting coefficients are preferred. This achieves a balanced optimization of pixel error and structural constraints.

[0011] Preferably, the For the bitrate optimization branch adapted to low-bandwidth power scenarios, the calculation formula is as follows: In the formula, For the real-time average coding bit rate of semantic streams, The real-time average coding bit rate for the pixel stream. , for , The weighting coefficients are preferred. , Prioritize compressing the lightweight semantic stream bitrate to ensure low-latency transmission during routine inspections. Through dual bitrate coordination constraints, the transmission bandwidth usage is minimized while ensuring service performance, making it suitable for low-bandwidth inspection scenarios.

[0012] Preferably, the specific implementation process of the retinal-like dual-guided attention enhancement mechanism in step 2 of the online encoding process for inspection videos is as follows: The first four convolutional layers of the ResNet-50 network are used as multi-scale basic feature extraction units. Redundant fully connected layers and global pooling layers are removed, while retaining the ability to extract shallow texture and deep semantic features. Multi-scale basic features are extracted from the input standardized video frames. The feature extraction calculation formula is as follows: ,in, To input power line inspection video frames, The output multi-scale basic feature map is 1 / 16 the size of the original image, and the number of feature channels is set to 2048. Simultaneously, the EfficientNet-B4 network is used to generate a 512×512 pixel-level saliency map of the power scene, accurately marking high-interest areas in the image; the Mask R-CNN model is used to segment targets, including conductors, insulators, line debris, and towers, generating a binary ROI mask map of power equipment; and dual-guided constraint signals are fused to generate an adaptive attention weight matrix, with the following feature weighting enhancement formula: Where ⊙ represents a channel-by-channel broadcast multiplication operation. This is the power target weighting adjustment coefficient, with a value ranging from 0.5 to 2.0, and a preferred value of 1.0. For saliency plot, This is the ROI mask image. This is used to enhance the attention-enhanced feature map, thereby strengthening the core power features and adaptively suppressing background redundancy.

[0013] Preferably, the semantic flow representation module in step 3 of the online encoding process for inspection videos is a semantic representation branch for power intelligent reasoning, employing a modified multi-scale void space pyramid pooling module. Addressing the inherent shortcomings of traditional ASPP modules—fixed void rate, single sensing field, and poor adaptability to irregularly shaped power targets—multiple sets of gradient-based adaptive sensing field combinations are configured to adapt to the multi-scale irregular features of slender power conductors, point insulators, and irregular line foreign objects. The multi-scale feature fusion calculation method is as follows: ,in, , , Three adaptive dilated convolution methods are used to optimize the dilation weights for slender power conductors, point insulators, and irregular foreign objects, respectively. For global pooling operations, adapted to the sparse semantic distribution of power; To optimize the multi-branch feature splicing and fusion strategy; For multi-branch feature channel compression convolution operation; This refined power semantic features effectively address the technical shortcomings of general models, such as incomplete extraction of features from irregularly shaped power targets and weak capture of contextual relationships.

[0014] Preferably, the pixel stream representation module in step 3 is a high-fidelity representation branch for manual power source tracing and verification, employing a four-level progressive processing logic of "image reconstruction - residual calculation - saliency modulation - lightweight coding": First, the final attention-enhanced feature map is processed using a multi-layer deconvolutional decoder. Upsampling and reconstruction are performed to obtain reconstructed video frames with the same size as the original video frames. Secondly, the refined pixel residuals are calculated, basic semantic redundancy is removed, and only high-frequency defect details are retained. The residual calculation formula is as follows: Combined with saliency plot Pixel-wise modulation of residual features differentiates and enhances details in the power target area while reducing background redundancy. The modulation formula is as follows: ,in, This is a pixel-by-pixel multiplication operation. The initial pixel residual features; To optimize and refine the residual map; finally, The input is a three-layer concatenated lightweight convolutional module, combined with an adaptive non-uniform quantization module to complete encoding compression. The pixel bitstream generation formula is: ,in, This is the quantized pixel bitstream. For lightweight convolutional coding modules, The dynamic step-size adaptive quantization module balances the need to preserve details of power equipment with the requirement of low bit rate compression, generating a high-fidelity pixel bitstream.

[0015] The coding method of this invention can fully cover the operation and maintenance perception task requirements of the power grid in all scenarios, including any one or more combined tasks such as detection of broken strands, wear and damage of transmission conductors, identification of cracked, contaminated, aged and missing states of insulators, detection of floating foreign objects obstructing transmission lines, monitoring of tower tilt and offset, identification of leakage and damage faults of transformers, and assessment of abnormal states of power distribution equipment.

[0016] Based on the same inventive concept, a lightweight semantic pixel joint coding system based on VCM is provided to implement the aforementioned coding method. The system includes: one or more processors, a readable storage device, a video acquisition terminal, and a network transmission module.

[0017] The readable storage device is used to store a computer program that can be executed by the processor, as well as power detection model parameters, ASPP module feature configuration parameters, attention weight parameters, encoding quantization parameters, and training configuration parameters; the video acquisition terminal is used to acquire power inspection time-series video streams in real time and complete pre-noise reduction and abnormal frame screening; the network transmission module is used to adaptively switch between single / dual bitstream transmission modes to achieve encrypted transmission of semantic bitstream and pixel bitstream; when the processor executes the computer program, it implements the processes of video preprocessing, retina-like attention enhancement, ASPP multi-scale feature purification, semantic-pixel dual-branch differential representation encoding, multi-objective joint optimization, and scene adaptive deployment.

[0018] The beneficial effects of this invention are as follows: Addressing the pain points specific to power grid inspection video scenarios, this invention innovatively designs a retina-like dual-guided attention enhancement mechanism, solving the problems of insufficient extraction of core power features and severe background interference in general models; it establishes a semantic-pixel dual-branch parallel coding architecture, simultaneously meeting the dual business needs of machine intelligent perception and manual traceability verification; and it innovatively constructs a ternary collaborative constraint loss function, breaking through the limitations of traditional single-index optimization in coding, achieving a balanced and optimal balance between perception accuracy, reconstruction fidelity, and bitrate overhead, significantly improving the scenario adaptability and engineering practicality of power grid video coding, and providing core technical support for the transmission and intelligent analysis of panoramic inspection videos for smart grids. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention;

[0020] Figure 2 This is a schematic diagram of the overall structure of the VCM encoding framework of the present invention. Detailed Implementation

[0021] Example 1 discloses a lightweight semantic-pixel joint coding method based on VCM, adaptable to the coding and intelligent sensing needs of power inspection videos in all scenarios such as drone inspection, robot inspection, and power grid online monitoring, breaking through the technical bottleneck of traditional coding that cannot simultaneously achieve perception, image quality, and bitrate. This invention adopts a complete engineering implementation process: first, a power inspection video dataset is constructed; then, end-to-end model training and optimization are completed based on a ternary collaborative constraint loss function; finally, feature extraction, retinal-like attention enhancement, and semantic-pixel dual-branch parallel coding inference are completed through the trained converged VCM model, achieving balanced optimization of multi-dimensional indicators. The specific complete implementation steps are as follows:

[0022] Step 1: Construct a multi-scenario semantic video dataset for power line inspection, and complete data preprocessing, annotation, augmentation, and segmentation: To address the industry pain points of poor adaptability of general video datasets to power scenarios, scarcity of irregular defect samples, and uneven sample distribution, which lead to insufficient feature extraction and weak generalization ability of general coding models, this invention independently constructs a multi-scale semantic video dataset for power grid inspection that is adapted to all power grid operation and maintenance scenarios. This dataset provides accurate data support for model training, feature learning, and coding optimization. The specific construction and preprocessing process is as follows: This dataset uses only real power operation and maintenance data, with no synthetic samples. The sample sources cover three main data collection scenarios: panoramic inspections by drones in the field, fixed-point inspections by robots at substations, and online monitoring terminals for transmission lines. It is adaptable to complex power grid environments such as mountainous areas, plains, suburbs, and indoor and outdoor substations. The dataset contains 22,000 short time-series video clips ranging from 2 to 5 seconds, with a uniform frame rate of 30fps, compatible with mainstream 1080P and 4K resolutions. Ultimately, over 260,000 valid video frames were extracted.

[0023] The dataset samples cover six core power equipment of the power grid and more than ten typical defect states, including broken strands, wear and damage of transmission conductors, pollution, aging, cracking and missing insulators, various floating foreign objects blocking the lines, tilting and offset of towers, leakage and damage of transformers and abnormalities of distribution equipment, etc. It also includes a large number of complex and difficult samples such as strong light and backlight, cloudy and foggy days, vegetation occlusion, and distant small targets, to comprehensively improve the robustness of the model in outdoor scenarios.

[0024] This invention employs a three-tiered fine-grained annotation strategy: pixel-level mask annotation, semantic label annotation, and image quality bitrate annotation. Unlike general video datasets, it performs pixel-level segmentation and annotation of the ROI of power equipment frame by frame, accurately locking the equipment outline region; it matches the semantic labels of defect categories and equipment status to improve the textual association information of defects; and it simultaneously annotates encoding-specific parameters such as the original video bitrate, image quality score, and compression distortion type, providing multi-dimensional supervision signals for semantic coding, pixel reconstruction, and bitrate-constrained multi-task training.

[0025] After the data is labeled, sample cleaning and data augmentation are carried out to remove invalid samples such as black screens, overexposed, severely blurred, and without effective power targets. The number of samples is expanded by scene adaptation and enhancement strategies such as random cropping, horizontal flipping, brightness fine adjustment, Gaussian noise superposition, and temporal frame shuffling to balance the distribution of various defective samples and avoid model training bias problems.

[0026] Finally, the dataset underwent global standardization preprocessing to unify input specifications: all video frames were fixedly scaled to 512×512 pixels to eliminate the impact of device resolution differences; ImageNet mean-variance was used for pixel normalization to stabilize training convergence; and adaptive Gaussian filtering was used to remove various noises caused by outdoor lighting and device electromagnetic interference. After preprocessing, the training, validation, and test sets were randomly divided in an 8:1:1 ratio to strictly ensure that the scene and defect distribution of the three datasets were consistent. This resulted in a standardized dataset that is temporally regular, sample-balanced, and adapted to the power coding task, for subsequent model iteration training.

[0027] Step 2: Build the overall VCM coding network architecture and initialize the basic model parameters: Based on the core technical solution of this invention, a complete VCM (Retina-like Joint Semantic Video Coding) network architecture is constructed. The architecture mainly comprises four core modules: a retina-like dual-guided attention enhancement module, a modified multi-scale ASPP semantic flow representation module, a four-level progressive pixel flow representation module, and a ternary collaborative constraint loss optimization module. The specific architecture initialization and parameter configuration are as follows: First, the basic feature extraction unit is initialized. The first four convolutional layers of the ResNet-50 network are used as the multi-scale basic feature extraction unit. Redundant fully connected layers and global pooling layers in the back end of the network are removed, fully preserving the network's ability to extract shallow texture details and capture deep semantic features. This avoids the parameter redundancy and increased computational load caused by fully connected layers, while also adapting to the multi-scale feature extraction requirements of power equipment. The basic feature extraction calculation formula is as follows: ),in, This is the current power inspection video frame input. The output multi-scale basic feature map is 1 / 16 the size of the original image, and the number of feature channels is set to 2048 to ensure multi-dimensional feature representation capability.

[0028] Next, the dual-guided feature generation unit is initialized, and the EfficientNet-B4 network is configured to generate a visual saliency map of the power scene, outputting a 512×512 pixel-level power scene saliency map. The Mask R-CNN segmentation model is initialized to lock the core target regions, including conductors, insulators, line debris, and towers, generating a binary ROI mask map of the power equipment. Finally, the semantic flow improved ASPP module, pixel flow deconvolution decoding, and lightweight encoding modules are initialized to complete the overall network parameter initialization and build a complete and trainable VCM encoding framework.

[0029] Step 3: Construct a three-factor collaborative loss function for perception, reconstruction, and bitrate, and complete end-to-end training and iterative optimization of the model. This step is the core training stage of the model. Addressing the technical contradictions of traditional video coding loss functions—namely, their single optimization dimension and the mutual constraints between perceptual accuracy, reconstructed image quality, and bitrate overhead—a ternary collaborative constraint loss function is constructed. The training set partitioned in step 2 is used to perform end-to-end collaborative iterative training of the global network parameters of the VCM coding framework. Network weights are continuously updated through backpropagation of the loss function to achieve balanced optimal convergence across multiple performance metrics. The specific training process is as follows:

[0030] The total loss function of this invention integrates three major branches: power sensing task loss, pixel reconstruction loss, and dual-stream collaborative bit rate constraint loss, breaking through the limitations of traditional single-loss optimization. The overall total loss calculation formula is as follows: In the formula, This represents the total end-to-end iterative loss of the model. The loss in power sensing tasks is used to constrain the accuracy of power equipment defect detection and condition identification. This is the pixel reconstruction loss, used to constrain the quality of video frame pixel detail and device structure restoration. This is a dual-stream collaborative bit rate constraint loss, used to manage the bandwidth overhead of semantic stream and pixel stream encoding transmission; It is an adaptive weight balancing coefficient that can be dynamically adjusted.

[0031] Based on the core business requirements of power operation and maintenance, namely "sensing first, image quality second, and extreme speed control", this embodiment prefers fixed values. Prioritize ensuring the accuracy of power defect detection and equipment identification, while also taking into account video reconstruction effects and low bitrate compression performance.

[0032] The power sensing task loss A loss branch is specifically designed for core power operation and maintenance tasks, integrating the dual task constraints of conductor defect detection and insulator status classification, and is specifically adapted to the differentiated perception needs of the power industry. The calculation formula is as follows: In the formula, the weighting coefficient is preferred. Priority is given to optimizing the detection accuracy of minor defects such as broken strands, wear, and damage in power transmission conductors; A dedicated combined loss for wire defects is developed, which integrates CIoU bounding box regression loss and cross-entropy classification loss to precisely constrain the localization accuracy and recognition accuracy of subtle defects. For insulator condition-specific classification loss, a cross-entropy loss function is adopted to constrain the classification accuracy of insulator conditions such as cracking, contamination, aging, and missing parts, which is fully adapted to the differentiated sensing tasks of power equipment.

[0033] The pixel reconstruction loss To balance the dual constraints of pixel accuracy and structural integrity in the loss branch, and to ensure consistency between the pixel fidelity of video frames and the critical structures of power equipment, the calculation formula is as follows: In the formula, the weighting coefficient is preferred. This achieves a balanced optimization of pixel error and structural constraints; The mean squared error loss is used to constrain the pixel grayscale deviation between the original video frame and the reconstructed video frame, ensuring the basic pixel accuracy of the image. It serves as a structural similarity index, used to constrain the consistency of key structural features such as edges, textures, and contours of power equipment, avoiding equipment structural distortion caused by encoding compression, and ensuring the reliability of fault tracing images.

[0034] The dual-stream bit rate constraint loss To adapt to the low bandwidth and weak network scenarios in power field applications, a dedicated bitrate optimization branch is used to collaboratively manage the transmission overhead of dual bitstreams. The calculation formula is as follows: In the formula, the weighting coefficient is preferred. Prioritize compressing the lightweight semantic stream bitrate to ensure low-latency, low-stuttering transmission during routine inspections; For the real-time average coding bit rate of semantic streams, The real-time average encoding bit rate of the pixel stream is achieved by using dual bit rate coordination constraints to minimize transmission bandwidth usage.

[0035] The model training employs an end-to-end iterative optimization strategy, based on the aforementioned standardized power line inspection dataset. The AdamW optimizer is selected, with an initial learning rate of 1e-4. A cosine annealing learning rate decay strategy is used, with a batch size of 16 and a total of 120 iterations. During training, each iteration calculates the global loss value using a ternary collaborative loss function, updates all network parameters via backpropagation, and uses three key metrics—monitoring and sensing accuracy, reconstructed SSIM value, and average bitrate—to validate the model. The optimal weight model is then saved, completing the convergence training of the VCM coding framework.

[0036] Step 4: Construct a retinal-like dual-guided attention enhancement mechanism to achieve adaptive enhancement of core electrical features: After model training converges, the inference and application phase begins. Addressing the technical pain points of general feature extraction models, such as poor adaptability to irregularly shaped power targets, easy loss of core features, and severe background redundancy interference, this invention simulates the visual characteristics of the human retina—focusing on key areas and weakening the background—by fusing visually salient features with power-specific target mask features. It adaptively generates an attention weight matrix to accurately enhance core power target features and suppress ineffective background redundancy features. The specific inference process is as follows: First, the first four layers of the trained ResNet-50 network are used to extract multi-scale basic features from the preprocessed video frames. Secondly, the trained EfficientNet-B4 network generates a 512×512 pixel-level saliency map of the power scene. It accurately labels high-interest areas in the image; at the same time, it uses the Mask R-CNN model to generate binary ROI mask images of power equipment. It can pinpoint key target areas such as conductors, insulators, foreign objects, and towers.

[0037] Finally, the dual-guided constraint signals are fused to adaptively generate the attention weight matrix and complete feature enhancement. The feature weighted enhancement formula is as follows: ,in, To enable channel-by-channel broadcast multiplication operations, precise fusion of weighted features and basic features is achieved; The value is the power target weight adjustment coefficient, ranging from 0.5 to 2.0. In this embodiment, the preferred value is 1.0, which balances the guiding weights of significance features and ROI mask features. For saliency plot, This is the ROI mask image. This is used to enhance the output attention-based feature map. Through this mechanism, the core power target features are adaptively enhanced, and background redundancy features are adaptively suppressed, providing high-quality feature input for subsequent dual-branch coding.

[0038] It should be understood that the ResNet-50 basic feature extraction network used in this step can be replaced by other suitable lightweight backbone networks. The replacement network should meet the following core features: have multi-scale feature extraction capabilities, and be able to output shallow texture features and deep semantic features simultaneously; be able to adapt to the feature extraction of power irregular targets and support irregular input feature mapping; and be able to flexibly remove redundant fully connected layers and pooling layers to meet the lightweight coding requirements of this invention.

[0039] Step 5: Build a semantic-pixel dual-branch parallel encoding architecture to achieve differentiated encoding and inference for dual services:

[0040] To address the technical shortcomings of traditional single-stream coding architectures in simultaneously meeting the dual requirements of machine intelligent perception and manual traceability verification, this invention constructs a semantic-pixel dual-branch parallel coding architecture. The attention-enhanced features obtained in step 4 are input into two independently trained representation modules. Lightweight semantic coding is achieved through the semantic stream to adapt to intelligent perception tasks, while high-fidelity pixel coding is achieved through the pixel stream to adapt to manual traceability tasks. The dual-branch parallel operation is independent of each other, accurately adapting to bidirectional power business scenarios. The specific reasoning process is as follows:

[0041] The semantic flow representation module is a lightweight semantic representation branch for power intelligent reasoning, employing a modified multi-scale void spatial pyramid pooling (ASPP) module. Addressing the shortcomings of traditional ASPP modules, such as fixed void ratio, single receptive field, and poor adaptability to irregular power line targets, this invention configures multiple sets of gradient-based adaptive receptive field combinations to accurately adapt to the multi-scale irregular features of slender power conductors, point insulators, and irregular line foreign objects. The multi-scale feature fusion calculation method is as follows: ,in, , , Three types of adaptive dilated convolution are used to optimize the dilution weights for slender power conductors, point insulators, and irregular line foreign objects, respectively. For global pooling operations, adapted to the sparse semantic distribution of power; To optimize the multi-branch feature splicing and fusion strategy; This involves multi-branch feature splicing and fusion, and channel compression convolution operations. This results in refined, concise electrical semantic features. The semantic stream module performs feature refinement, dimensionality compression, and adaptive quantization, ultimately generating a lightweight semantic codestream. This ensures the accuracy of backend intelligent sensing while reducing transmission overhead. Figure 2 As shown, this module incorporates multiple cascaded Transformer modules to refine multi-scale contextual semantic representation features. Each Transformer module is followed by a rate control switch to adaptively adjust the transmission rate of the semantic bitstream.

[0042] The pixel stream representation module is a high-fidelity representation branch for manual power source tracing and verification. It adopts a four-level progressive processing logic of "image reconstruction - residual calculation - saliency modulation - lightweight coding", preserving subtle defects and texture details of the equipment throughout the process. The specific process is as follows: First, attention-enhanced feature maps are obtained through a multi-layer deconvolutional decoder. Upsampling and reconstruction are performed to obtain a reconstructed image with the same size as the original video frame. Secondly, the refined pixel residuals are calculated, basic semantic redundancy is removed, and only high-frequency defect details are retained. The residual calculation formula is as follows: ,in, The initial pixel residual features accurately characterize the detailed differences between the original and reconstructed images. Furthermore, a saliency map is introduced to modulate the residual features pixel-by-pixel, differentially enhancing the details of the power target and weakening background redundancy. The modulation formula is: ,in, To refine the residual plot after optimization. Finally, The input is a three-layer concatenated lightweight convolutional module, combined with an adaptive non-uniform quantization module to complete encoding compression. The pixel bitstream generation formula is: ,in, This is the quantized pixel bitstream. It is a three-layer cascaded lightweight convolutional coding module. It is a dynamic step-size adaptive quantization module that balances detail preservation and low bit rate compression to generate high-fidelity pixel bit streams, supporting manual fault tracing and verification.

[0043] Furthermore, the coding method proposed in this invention can fully cover all power grid operation and maintenance perception tasks, including any one or more combined tasks such as detection of broken strands, wear, and damage defects in transmission conductors, identification of cracked, contaminated, aged, and missing states of insulators, detection of floating foreign objects obstructing transmission lines, monitoring of tower tilt and offset, identification of transformer leakage and damage faults, and assessment of abnormal states of power distribution equipment. It has extremely strong scenario adaptability and engineering versatility.

[0044] Example 2: Based on the same inventive concept, the present invention also discloses a lightweight semantic pixel joint coding system based on VCM, used to implement the lightweight semantic pixel joint coding method based on VCM described in Example 1. The system specifically includes: one or more processors, a readable storage device, a video acquisition terminal, and a network transmission module.

[0045] The readable storage device stores the computer program that can be executed by the processor, as well as power detection model parameters, ASPP module feature configuration parameters, attention weight parameters, encoding quantization parameters, and training configuration parameters, providing data and model support for the entire system operation. The video acquisition terminal is used to acquire power inspection time-series video streams transmitted by drones, inspection robots, and online monitoring equipment in real time, and simultaneously complete pre-noise reduction and preliminary screening of abnormal frames to ensure the quality of input data. The network transmission module is used to adaptively switch between single / dual bitstream transmission modes, dynamically prioritizing the transmission of semantic bitstream or simultaneously transmitting dual bitstreams according to the on-site bandwidth environment, to achieve encrypted and stable transmission of semantic bitstream and pixel bitstream. When the processor executes the computer program stored in the readable storage device, it fully realizes the entire process of power dataset construction, video preprocessing, model training optimization, retina-like attention enhancement, dual-branch differential encoding, and adaptive bitstream transmission.

[0046] Since the system described in this embodiment is a hardware architecture for implementing the encoding method of Embodiment 1 of the present invention, those skilled in the art can clearly understand the specific structure, module adaptation method, and modified extension form of the system based on the technical solution described in Embodiment 1 of the present invention, and therefore will not be repeated here. All system structures and hardware extension forms adapted to the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0047] Example 3: Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, can implement all the steps of the lightweight semantic pixel joint encoding method based on VCM described in Example 1.

[0048] Those skilled in the art will clearly understand that this computer-readable medium is compatible with all the logic, parameter configurations, and optimization strategies of the core encoding method of this invention, enabling the entire process of constructing a dedicated power inspection dataset, video preprocessing, iterative training of the model using ternary loss, attention feature enhancement, dual-branch parallel encoding, and adaptive bitstream transmission. Based on the technical content of Embodiment 1 of this invention, those skilled in the art can flexibly adapt different types of computer-readable storage media for program deployment and application without creative effort; therefore, specific deployment details will not be elaborated further. Any computer-readable medium capable of implementing the core encoding method of this invention falls within the protection scope of this invention.

[0049] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A lightweight semantic pixel joint encoding method based on VCM, characterized in that, This includes the offline training and convergence process for the model, and the online encoding process for inspection videos. (1) Model offline training convergence process: Construct a fully labeled power inspection dataset and preprocess it, build a VCM coding network including a retina-like dual-guided attention enhancement module, a semantic stream representation module, and a pixel stream representation module; construct a perception-reconstruction-bitrate ternary collaborative constraint loss function, input standardized power video frames into the network to perform feature enhancement and dual-branch coding to obtain dual-stream bitstreams, calculate the total loss of the ternary collaborative constraint, and iteratively update all network parameters through backpropagation until the model's perception accuracy, reconstruction fidelity, bitrate overhead, and other performance indicators converge in a balanced manner, and save the optimal coding model; (2) Online encoding process for inspection videos: Step 1: Real-time acquisition of power inspection video stream, reading time-series video images frame by frame, completing size standardization, pixel normalization and noise filtering, removing invalid abnormal frames such as black screen, overexposure, and severe blur, and generating standardized video frames after the above preprocessing. Step 2: Based on the retina-like dual-guided attention enhancement mechanism, the standardized video frames are processed. Multi-scale basic features of the standardized video frames are extracted through a lightweight backbone network. The image visual saliency features and power equipment-specific ROI mask features are fused to generate an adaptive attention weight matrix. This completes the enhancement of core power target features and the suppression of background redundancy features, resulting in attention-enhanced features. Step 3: The attention-enhancing features are processed synchronously using a semantic-pixel dual-branch parallel VCM coding framework. The attention-enhancing features are input into the semantic stream representation module and the pixel stream representation module respectively. The semantic representation features are purified by the multi-scale context enhancement module built into the semantic stream representation module, and a lightweight semantic code stream is generated by dimensionality compression and adaptive quantization. The attention-enhanced features are reconstructed using a pixel stream representation module, and a high-fidelity pixel stream is generated through lightweight encoding and quantization. Step 4: Adaptive transmission bitstream. Switch between single-stream or dual-stream encrypted transmission based on the on-site network bandwidth. In low-bandwidth environments, only the semantic bitstream is transmitted. When bandwidth is sufficient, both the semantic bitstream and the pixel bitstream are transmitted simultaneously.

2. The lightweight semantic pixel joint coding method based on VCM according to claim 1, characterized in that, The ternary collaborative constraint loss function adopts a multi-objective joint optimization loss model, and the overall total loss calculation formula is as follows: In the formula, This represents the total end-to-end iterative loss of the model. The loss in power sensing tasks is used to constrain the accuracy of power equipment defect detection and condition identification. This is the pixel reconstruction loss, used to constrain the pixel detail and structural restoration quality of video frames; This is a dual-stream collaborative bit rate constraint loss, used to manage the bandwidth overhead of semantic stream and pixel stream encoding transmission; It is an adaptive weight balancing coefficient that can be dynamically adjusted.

3. The lightweight semantic pixel joint coding method based on VCM according to claim 2, characterized in that, The The calculation formula integrates the dual-task constraints of defect detection and equipment status classification: In the formula, A dedicated loss function for conductor defects is used, which integrates CIoU bounding box regression loss and cross-entropy classification loss. For insulator state-specific classification loss, the cross-entropy loss function is used to constrain classification accuracy. , for , The weighting coefficients.

4. The lightweight semantic pixel joint coding method based on VCM according to claim 2, characterized in that, The The loss branch, which balances pixel accuracy and device structural integrity, is calculated using the following formula: In the formula, The mean squared error loss is used to constrain the pixel grayscale deviation between the original video frame and the reconstructed video frame. It is a structural similarity index used to constrain the consistency of edges, textures, and contours of power equipment. , for , The weighting coefficients.

5. The lightweight semantic pixel joint coding method based on VCM according to claim 2, characterized in that, The For the bitrate optimization branch adapted to low-bandwidth power scenarios, the calculation formula is as follows: In the formula, For the real-time average coding bit rate of semantic streams, The real-time average coding bit rate for the pixel stream. , for , The weighting coefficients.

6. The lightweight semantic pixel joint coding method based on VCM according to claim 1, characterized in that, The specific implementation process of the retinal-like dual-guided attention enhancement mechanism described in step 2 of the online encoding process for inspection videos is as follows: The first four convolutional layers of the ResNet-50 network are used as multi-scale basic feature extraction units. Redundant fully connected layers and global pooling layers are removed, while retaining the ability to extract shallow texture and deep semantic features. Multi-scale basic features are extracted from the input standardized video frames. The basic feature extraction formula is as follows: ),in, To input power line inspection video frames, The output multi-scale basic feature map is 1 / 16 the size of the original image, and the number of feature channels is set to 2048. Simultaneously, an EfficientNet-B4 network is used to generate a 512×512 pixel-level saliency map of the power scene, marking high-interest areas in the image; the Mask R-CNN model is used to segment the target, including conductors, insulators, line debris, and towers, generating a binary ROI mask map of power equipment; and dual guided constraint signals are fused to generate an adaptive attention weight matrix. The core feature enhancement formula is as follows: Where ⊙ represents a channel-by-channel broadcast multiplication operation. This is the power target weighting adjustment coefficient, with a value ranging from 0.5 to 2.

0. For saliency plot, This is the ROI mask image. This is used to enhance the attention-enhanced feature map, thereby strengthening the core power features and adaptively suppressing background redundancy.

7. The lightweight semantic pixel joint coding method based on VCM according to claim 1, characterized in that, The semantic flow representation module mentioned in step 3 of the online encoding process for inspection videos is a semantic representation branch for power intelligent reasoning. It adopts a modified multi-scale hollow spatial pyramid pooling module; it is configured with multiple sets of gradient-based adaptive receptive field combinations to adapt to multi-scale irregular features. The multi-scale feature fusion calculation method is as follows: ,in, , , Three adaptive dilated convolution methods are used to optimize the dilation weights for slender power conductors, point insulators, and irregular foreign objects, respectively. For global pooling operations, adapted to the sparse semantic distribution of power; To optimize the multi-branch feature splicing and fusion strategy, For multi-branch feature channel compression convolution operation, This refers to the refined and purified semantic features of the power industry.

8. The lightweight semantic pixel joint coding method based on VCM according to claim 1, characterized in that, The pixel stream representation module in step 3 of the online encoding process for inspection videos is a high-fidelity representation branch for manual power source tracing and verification, employing a four-level progressive processing logic of image reconstruction, residual calculation, saliency modulation, and lightweight encoding. First, attention-enhanced feature maps are processed using a multi-layer deconvolutional decoder. Upsampling and reconstruction are performed to obtain reconstructed video frames with the same size as the original video frames. Secondly, the refined pixel residuals are calculated, basic semantic redundancy is removed, and only high-frequency defect details are retained. The residual calculation formula is as follows: Combined with saliency plot Pixel-wise modulation of residual features differentiates and enhances details in the power target area while reducing background redundancy. The modulation formula is as follows: ,in, This is a pixel-by-pixel multiplication operation. For the initial pixel residual features, To optimize and refine the residual map; finally, The input is a three-layer concatenated lightweight convolutional module, combined with an adaptive non-uniform quantization module to complete encoding compression. The pixel bitstream generation formula is: ,in, This is the quantized pixel bitstream. For lightweight convolutional coding, This is a dynamic step-size adaptive quantization module.

9. A lightweight semantic pixel joint coding system based on VCM, characterized in that, The system for implementing the lightweight semantic pixel joint coding method based on VCM as described in any one of claims 1-8 includes: one or more processors, a readable storage device, a video acquisition terminal, and a network transmission module; The readable storage device is used to store a computer program that can be executed by the processor, as well as power detection model parameters, ASPP module feature configuration parameters, attention weight parameters, encoding quantization parameters, and training configuration parameters; the video acquisition terminal is used to acquire power inspection time-series video streams in real time and complete pre-noise reduction and abnormal frame screening; the network transmission module is used to adaptively switch between single / dual bitstream transmission modes to achieve encrypted transmission of semantic bitstream and pixel bitstream; when the processor executes the computer program, it implements the processes of video preprocessing, retina-like attention enhancement, ASPP multi-scale feature purification, semantic-pixel dual-branch differential representation encoding, multi-objective joint optimization, and scene adaptive deployment.

10. A computer-readable medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the lightweight semantic pixel joint encoding method based on VCM as described in any one of claims 1-8.

Images