Multi-level visual coding and intelligent analysis method based on CLIP semantic supervision

Abstract

The application provides a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision, relates to the technical field of computer vision, and comprises the following steps: collecting power scene videos, encoding the videos, reserving key image elements, generating preliminary structure, texture and semantic features, inputting a joint encoder, fusing uniform feature representations after hierarchical processing by a multi-channel compression framework, and outputting compressed feature data suitable for power scene transmission bandwidth and intelligent analysis requirements; taking a pre-trained CLIP model as a semantic supervision source, performing semantic supervision training on the compressed feature data to optimize semantic consistency; performing conditional decoding on the optimized data, dynamically generating corresponding views according to a preference vector, synchronously analyzing semantic information, and outputting intelligent analysis results containing power equipment state labels and defect region coordinates; and the application can reduce power scene transmission and storage costs, improve machine analysis efficiency and cross-scene generalization ability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision. Background Technology

[0002] Visual monitoring systems in the power industry bear the core functions of power grid equipment status perception and fault early warning. However, their application scenarios are characterized by significant environmental specificities and resource constraints: transmission lines are mostly laid along mountainous, hilly, and Gobi desert areas, and monitoring equipment often faces extreme conditions such as heavy rain, icing, and strong electromagnetic interference, resulting in frequent 4G / 5G signal blockage and poor bandwidth stability (average bandwidth fluctuation reaches 30%-50%). Although substations are located on the outskirts of towns, they need to transmit 16-32 channels of high-definition video (25fps / 1080P specification per channel) from main transformers, switch rooms, and capacitor banks simultaneously, resulting in bandwidth resources being saturated for a long time. Distribution substations are characterized by "numerous points and wide coverage," with monitoring points scattered in urban-rural fringe areas, and limited storage capacity at edge nodes (the upper limit of storage for a single device is mostly 1TB), making it difficult to support the retention of all video data. Against this backdrop, the "efficient transmission, low-cost storage, and rapid analysis" of power visual data needs to break through the bottlenecks of traditional encoding technologies, and existing solutions are no longer suitable for the core needs of the power grid's "digital transformation and intelligent upgrading."

[0003] The core objective of visual coding technology for power scenarios is to reduce data redundancy while retaining key features required for machine vision analysis (such as equipment defect detection and foreign object identification) under limited bandwidth and storage resources. Its technological evolution has undergone three generations of iterations: The first generation was based on traditional pixel-level compression technology, which reduced redundancy through intra-frame prediction and inter-frame motion compensation. While this could meet human visual viewing needs, a single 1080P video still required 2-4 Mbps bandwidth, and a substation's daily video data volume reached 50-80 GB, with annual storage costs exceeding 100,000 yuan. The second generation is based on sparse sensing digital retina technology, simulating the biological vision's characteristic of "focusing on significant changes," only processing moving areas (such as line swaying caused by icing) or areas of structural change (such as insulation). The first generation of compression technology, which uses deep learning-based task-driven compression, extracts device-specific features (such as transformer oil level and cable joint temperature) through CNN / Transformer models. While this improves the adaptability of machine analysis, the model's generalization ability is limited, supporting only a single device or a single defect type. When switching to scenarios such as "foreign object detection in transmission lines" or "meter identification in distribution substations," hundreds to thousands of samples need to be re-labeled and retrained end-to-end, with an adaptation cycle of 2-4 weeks, which is difficult to meet the power grid's requirements for "second-level response and dynamic adaptation."

[0004] Therefore, there is an urgent need for a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision, which can reduce transmission and storage costs while improving machine analysis efficiency and cross-scenario generalization ability, and truly achieve the goal of "one-time coding, multi-task reuse, and full-scenario adaptation" in the power industry.

[0006] The technical solution of this invention is as follows:

[0007] Firstly, this application provides a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision, which includes the following steps:

[0008] S1. Collect power scene videos, and generate preliminary structural features, texture features and semantic features by encoding the videos and retaining key image elements.

[0009] S2. Input the preliminary structural features, texture features and semantic features into the joint encoder, and after being processed in layers by the multi-channel compression framework, they are fused into a unified feature representation, and the compressed feature data adapted to the transmission bandwidth and intelligent analysis requirements of the power scenario is output.

[0010] S3. Using the pre-trained CLIP model as the source of semantic supervision, semantic consistency optimization is achieved by performing semantic supervision training on compressed feature data.

[0011] S4. Perform conditional decoding on the optimized compressed feature data, dynamically generate corresponding views based on the preference vector, synchronously parse semantic information, and output intelligent analysis results containing power equipment status labels and defect area coordinates.

[0012] It should be noted that the key image elements retained are regions with significant motion or structure.

[0013] Further, step S1 includes:

[0014] Using real-time video streams of power scenarios as input, the system performs three levels of processing: salient region identification, region acquisition and encoding, and preliminary feature generation.

[0015] The above-mentioned significant region determination adopts a two-dimensional determination mechanism of motion change rate and structural difference, and compares the current video frame with the previous frame frame by frame to select the region that needs to be collected and encoded.

[0016] The calculation process for the aforementioned rate of change of motion includes:

[0017]

[0018] In the formula, , represents a vector For vectors The norm;

[0019] The calculation process for the above structural dissimilarity includes:

[0020]

[0021] In the formula, It is a structural similarity index. For the current frame, For the previous frame, This represents the average local brightness. For local contrast, For local covariance, The stability coefficient;

[0022] The aforementioned preliminary feature generation includes the simultaneous generation of three types of preliminary features: structural features, texture features, and semantic features, while preserving key image elements of the device;

[0023] The aforementioned structural features employ the Sobel operator to calculate the gradient to obtain the gradient magnitude and direction. The calculation formula includes:

[0024]

[0025]

[0026] In the formula, For gradient magnitude, For horizontal gradient, For vertical gradient, The gradient direction;

[0027] The above texture features are extracted using multi-scale, multi-directional Gabor filter kernels to extract the surface texture and material features of the device. The calculation formula includes:

[0028]

[0029] ,

[0030] In the formula, ( () represents the original image pixel coordinates in the Cartesian coordinate system. To control the carrier wavelength of the filtering frequency, Aspect ratio, Let be the standard deviation of the Gaussian envelope. , The image pixel coordinates are adjusted to the Cartesian coordinate system. Through multi-scale and multi-directional combinations, Gabor filtering can accurately extract key texture features on the surface of power equipment (such as the texture of transformer heat sinks and the direction of insulator cracks), providing high-recognition input for subsequent defect detection.

[0031] The aforementioned semantic features are input into a CLIP visual encoder with frozen weights, after normalized salient region images. After 12 layers of Transformer encoding, the mean pooling result of the last layer's output is used as a 512-dimensional semantic vector. This effectively compresses image data in power vision analysis while retaining the semantic information required for equipment identification, providing high-quality feature input for subsequent tasks such as defect detection and condition monitoring.

[0032] Further, step S2 includes:

[0033] Two-dimensional discrete wavelet transform (DWT) is used to perform multi-scale decomposition of structural features into low-frequency and high-frequency coefficients. The low-frequency coefficients containing the main contour information are retained, and the high-frequency coefficients are thresholded to remove redundant details, and the low-frequency coefficients of the structural features are output. The threshold is set to 0.02 × the maximum high-frequency coefficient.

[0034] The texture direction and frequency features are extracted by Gabor filtering. K-means clustering is used to group the texture feature vectors. The center vector of each cluster is retained as a representative texture feature. Duplicate texture data is reduced and the texture cluster center vector is output.

[0035] Construct a semantic vector matrix based on the 512-dimensional semantic vector generated in step S1. By calculating the covariance matrix And solve for the covariance matrix. Given the eigenvalues ​​and corresponding eigenvectors, perform PCA dimensionality reduction on the semantic vector matrix X to output a semantic embedding vector. The calculation process includes:

[0036]

[0037]

[0038] In the formula, Let covariance matrix be the variance matrix. Let X be the number of pixel blocks within the salient region, and let X be the semantic vector matrix. Let X be the mean vector. The semantic vector after dimensionality reduction. The projection matrix constructed from the eigenvectors;

[0039] The structural feature low-frequency coefficients, texture clustering center vectors, and semantic embedding vectors are concatenated and mapped to a unified feature representation through a fully connected layer, outputting compressed feature data that adapts to the transmission bandwidth and intelligent analysis requirements of power scenarios.

[0040] Furthermore, in step S3, the semantic consistency optimization mentioned above includes: global scale alignment, local scale alignment, and instance scale alignment;

[0041] The aforementioned global scale alignment inputs the original image feature data and compressed feature data into the CLIP semantically supervised visual encoder, extracts the global semantic embedding vector (dimension=512), calculates the cosine similarity between the two and uses it as the global supervision loss; among them, the visual encoder adopts CLIP-ViT-B / 16;

[0042] The aforementioned local scale alignment divides the original image feature data and compressed feature data into 16×16 local blocks, extracts the CLIP semantic vector of each local block, and calculates the mean square error of the corresponding local block semantic vector as the local supervision loss.

[0043] The above instance scale alignment is used to mask and annotate power equipment instances, extract CLIP semantic vectors of corresponding instances from the original image feature data and compressed feature data, and calculate Euclidean distance as instance supervision loss.

[0044] Secondly, this application provides an electronic device, comprising:

[0045] Memory, used to store one or more programs;

[0046] processor;

[0047] When one or more of the above programs are executed by the above processor, a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision is implemented as described in any of the first aspects above.

[0048] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in any of the first aspects above.

[0049] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0050] (1) The present invention provides a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision. Through the multi-scale alignment mechanism of CLIP semantic supervision, the feature distribution offset is reduced, and the generalization accuracy of the model in unknown power scenarios and tasks is increased to more than 92%. Overfitting is avoided, and there is no need to retrain the model for different scenarios, which significantly improves the cross-scenario generalization ability.

[0051] (2) The present invention adopts digital retinal sensing acquisition and nested joint compression, only encodes the salient region and compresses the features in layers, which significantly reduces the video data volume and bandwidth consumption, and saves transmission and storage costs;

[0052] (3) The present invention retains structural, texture and semantic features in the encoding stage, providing relevant input for subsequent intelligent analysis, supporting multiple tasks such as equipment defect detection and status recognition, improving analysis efficiency and enhancing machine vision adaptability. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision according to the present invention;

[0055] Figure 2This is a schematic structural block diagram of a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision according to the present invention.

[0056] Figure 3 This is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.

[0057] Icons: 101, memory; 102, processor; 103, communication interface. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0063] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0064] Example 1

[0065] Please see Figure 1-2 , Figure 1-2 The figures shown are flowcharts and schematic structural diagrams of a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision provided in this application.

[0066] This application provides a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision, which includes the following steps:

[0067] S1. Collect power scene videos, and generate preliminary structural features, texture features and semantic features by encoding the videos and retaining key image elements.

[0068] S2. Input the preliminary structural features, texture features and semantic features into the joint encoder, and after being processed in layers by the multi-channel compression framework, they are fused into a unified feature representation, and the compressed feature data adapted to the transmission bandwidth and intelligent analysis requirements of the power scenario is output.

[0069] S3. Using the pre-trained CLIP model as the source of semantic supervision, semantic consistency optimization is achieved by performing semantic supervision training on compressed feature data.

[0070] S4. Perform conditional decoding on the optimized compressed feature data, dynamically generate corresponding views based on the preference vector, synchronously parse semantic information, and output intelligent analysis results containing power equipment status labels and defect area coordinates.

[0071] In a preferred embodiment, step S1 includes:

[0072] Using real-time video streams of power scenarios as input, the system performs three levels of processing: salient region identification, region acquisition and encoding, and preliminary feature generation.

[0073] The determination of significant regions adopts a two-dimensional determination mechanism of motion change rate and structural difference, and compares the current video frame with the previous frame frame by frame to select the regions that need to be collected and encoded.

[0074] The calculation process for the rate of change of motion includes:

[0075]

[0076] In the formula, , represents a vector For vectors The norm;

[0077] The calculation process for structural dissimilarity includes:

[0078]

[0079] In the formula, It is a structural similarity index. For the current frame, For the previous frame, This represents the average local brightness. For local contrast, For local covariance, The stability coefficient;

[0080] It should be noted that the determination of saliency regions adopts a joint determination rule: only regions that meet the "rate of change of motion standard" or "structural difference standard" are marked as "saliency regions," while non-saliency regions (such as static sky and unchanging vegetation) are directly discarded, which can initially reduce redundant data by 40%-50%. For any pixel (x, y) within a saliency region, the region acquisition coding uses its upper left corner as the reference point. ,Left ,superior Linear prediction is performed based on the gray values ​​of three neighboring pixels. Only the "residual between the actual value and the predicted value" is encoded to reduce redundancy. The specific formula is as follows:

[0081]

[0082] in, For any pixel Actual grayscale value (values ​​[0, 255]), The grayscale value is predicted; a=0.3, b=0.4, c=0.3 are the prediction coefficients optimized by the power scene samples (satisfying a+b+c=1 to ensure prediction stability); if the neighboring pixels exceed the region boundary, the grayscale value of the region boundary pixel is taken by default.

[0083] Preliminary feature generation includes the simultaneous generation of three types of preliminary features: structural features, texture features, and semantic features, while preserving key image elements such as device geometric contours and surface textures.

[0084] The structural features are extracted through a three-stage process: gradient calculation, non-maximum suppression, and dual threshold filtering, to obtain the device's geometric contour and edge information. Gradient calculation employs horizontal and vertical Sobel operators to calculate the gradient magnitude and direction. The calculation formula includes:

[0085]

[0086]

[0087]

[0088] In the formula, For gradient magnitude, For horizontal gradient, For vertical gradient, The gradient direction (used for non-maximum suppression, categorized into four levels: 0°, 45°, 90°, and 135°, covering the longitudinal and transverse texture directions of the cable sheath);

[0089] Dual threshold filtering sets the lower threshold. =50, Upper limit threshold =150, the gradient magnitude G is filtered, and the final output is a binary edge map E (edge ​​pixel value 255, non-edge 0), ensuring that the key structural information of the device is not lost during the subsequent compression process.

[0090] Texture features were extracted from the device surface using 24 Gabor filter kernels across 6 scales and 4 directions. The calculation formula includes:

[0091]

[0092] ,

[0093] In the formula, ( () represents the original image pixel coordinates in the Cartesian coordinate system. To control the carrier wavelength of the filtering frequency, Aspect ratio (enhancing orientation sensitivity). The standard deviation of the Gaussian envelope (matching the 7×7 filter kernel size), , The image pixel coordinates are adjusted to the Cartesian coordinate system. Through multi-scale and multi-directional combinations, Gabor filtering can accurately extract key texture features on the surface of power equipment (such as the texture of transformer heat sinks and the direction of insulator cracks), providing high-recognition input for subsequent defect detection.

[0094] By convolving the data with 24 Gabor filter kernels across 6 scales and 4 directions, 24 texture response maps are obtained. The 24-dimensional texture feature vector that forms this block. ;in (This is the set of pixel values ​​of the k-th response map within pixel block b), enabling dimensional standardization and region representativeness extraction of texture features.

[0095] The semantic features are generated by inputting the normalized salient region image into the CLIP visual encoder with frozen weights. After 12 layers of Transformer encoding, the mean pooling result of the last layer's output is taken as the 512-dimensional semantic vector, as shown in the formula:

[0096]

[0097] in, (The vector dimension is 512). This vector contains semantic information about the power equipment category (such as the unique semantic representation of "10kV transformer" and "suspension insulator"). In power visual analysis, it can effectively compress image data while retaining the semantic information required for equipment identification, providing high-quality feature input for subsequent tasks such as defect detection and condition monitoring.

[0098] In a preferred embodiment, step S2 includes:

[0099] Format standardization: Structural features (edge ​​maps) are converted to single-channel 8-bit image format, texture features (24-dimensional vectors) are converted to floating-point arrays (precision float32), and semantic features (512-dimensional vectors) are converted to float32 arrays. All features are uniformly labeled using the format "Region ID + Feature Type + Data Length + Feature Data". Anomaly filtering: The mean gradient magnitude of structural features is calculated. Regions with a mean value < 10 (indistinct edges) are marked as "low-value regions," and their texture and semantic features are temporarily stored locally (not involved in compressed transmission). When the L2 norm of the semantic feature vector is < 50 (sparse semantic information), it is re-extracted (to avoid encoding invalid semantics).

[0100] Three-channel layered compression:

[0101] Structural feature channel compression: Select the "Haar wavelet basis" and use two-dimensional discrete wavelet transform (DWT) to perform multi-scale decomposition of structural features into low-frequency coefficients LL1 (accounting for 25%, including core contours) and high-frequency coefficients LH1 (horizontal details), HL1 (vertical details), and HH1 (diagonal details). Perform a second DWT decomposition on LL1 to obtain LL2 (accounting for 6.25%, the core of the core contour) and LH2, HL2, and HH2. Set high-frequency coefficient thresholds: 0.025 × maximum high-frequency coefficient for power transmission scenarios (more details need to be retained for conductor contours), and 0.018 × maximum high-frequency coefficient for substation scenarios (requires simplification due to dense equipment). High-frequency coefficients below the threshold are set to zero. Run-length encoding (RLE) is used to compress the quantized coefficients.

[0102] Texture feature channel compression: Gabor filtering is used to extract texture direction and frequency features. K-means clustering (K=8 clusters) is used to group the texture feature vectors, retaining the center vector of each cluster as a representative texture feature to reduce duplicate texture data. A texture vector set is constructed using a 24-dimensional texture vector T as input. (M is the number of pixel blocks within the salient region, with each block consisting of 16×16 pixels); K-means clustering algorithm is used for... Clustering, with cluster number K=8 (verified by power equipment texture samples, K=8 achieves a redundancy reduction rate of 65% and a texture recall rate ≥92%), the cluster center update formula is:

[0103]

[0104] For the first Class 1 Cluster centers in the next iteration For the first In the nth iteration A sample set of classes The iteration stops when the change in cluster centers is ≤0.001 (the number of samples).

[0105] Construct a semantic vector matrix based on the 512-dimensional semantic vector generated in step S1. By calculating the covariance matrix And solve for the covariance matrix. eigenvalues With corresponding feature vectors The first 128 principal components (cumulative contribution rate ≥ 95%, ensuring semantic information loss ≤ 5%) were selected to construct the projection matrix. PCA dimensionality reduction is performed on the semantic vector matrix X to output semantic embedding vectors. The calculation process includes:

[0106]

[0107]

[0108] In the formula, Let covariance matrix be the variance matrix. Let X be the number of pixel blocks within the salient region, and let X be the semantic vector matrix. Let X be the mean vector. The semantic vector after dimensionality reduction. The projection matrix constructed from the feature vectors compresses the semantic feature dimension from 512 dimensions to 128 dimensions, reducing the data volume by 75%.

[0109] Feature fusion and unified encoding: Low-frequency coefficients of structural features, texture cluster center vectors, and semantic embedding vectors are concatenated and mapped to a unified feature representation through a fully connected layer (number of neurons = 256). This unified feature representation is then processed through a fully connected layer with 256 neurons. The system performs linear transformation and activation, and then maps the output dimension to the target dimension through a fully connected layer with 64 output neurons, outputting compressed feature data that adapts to the transmission bandwidth and intelligent analysis requirements of the power scenario.

[0110] As a preferred implementation, in step S3, semantic consistency optimization includes: global scale alignment, local scale alignment, and instance scale alignment; in this embodiment of the invention, global scale, local and instance scale alignment are achieved through three levels of semantic matching to ensure semantic consistency of the whole, details, and device instances, respectively, and to avoid feature distribution shift after compression.

[0111] Global scale alignment will adjust the original image Reconstructed image with output from joint encoder The inputs are fed into the CLIP visual encoder (using CLIP-ViT-B / 16, relying on its pre-trained cross-modal semantic capabilities), and after passing through 12 layers of Transformer encoding (each layer contains 12 attention heads, and the hidden layer dimension is 768), the mean pooling result of the last layer output is taken to obtain a 512-dimensional global semantic vector. (Original image) and (Image reconstruction) The vector dimensions are kept consistent with the CLIP pre-trained output to ensure semantic comparability. The cosine similarity between the two is calculated and used as the global supervision loss. The semantic overlap is measured by the dot product of the normalized vectors, as shown in the formula:

[0112] in The L2 norm ensures that vectors of different scales can be directly compared, and the closer the result is to 1, the more consistent the semantics.

[0113] Global Supervision of Loss and Objectives: Defining Loss The training objective is to minimize this loss using gradient descent. (Corresponding semantic similarity > 90%), to ensure that the global semantics of core scenarios and equipment such as "substation-transformer" and "transmission line-insulator" are not lost;

[0114] Local scale alignment: aligning the original image With reconstructed image All local blocks are divided into 16×16 pixel sizes (adapting to CLIP's patch division method), resulting in a total of 1920×1080 resolution images. A block, denoted as (The i-th block of the original image) and (Reconstruct the i-th block of the image) Extract the CLIP semantic vector for each local block: Each local block is individually input into the local feature branch of the CLIP visual encoder (local detail capture is enhanced through a window attention mechanism), and the output is a 512-dimensional vector. (Original image) and (Reconstructing the image); Calculate the mean squared error (MSE) of the semantic vector of the corresponding local block as the local supervision loss: Measure the semantic difference of a single block using MSE, and take the average of all blocks as the local loss:

[0115]

[0116]

[0117] Training objectives (Local semantic error < 5%), ensuring that the semantic features of local details of the device are not destroyed by the compression process;

[0118] Instance semantic vector extraction: extracting semantic vectors from the original image based on the mask. With reconstructed image Example of a cropping device: inputting the CLIP visual encoder yields a 512-dimensional vector. (The j-th instance of the original image) and (The j-th instance of the reconstructed image) ;

[0119] Euclidean distance and instance loss: The semantic difference of a single instance is measured by Euclidean distance, and the average value is taken as the instance loss.

[0120]

[0121]

[0122] Training objectives <2.0) (Based on sample verification, this threshold ensures that the semantic confusion rate of device instances is <3%), avoiding instance-level semantic misalignment.

[0123] The total supervisory loss is the weighted sum of the three factors (with weights of 0.4, 0.3, and 0.3 respectively), as shown in the formula:

[0124]

[0125] During training, the CLIP semantic supervision pre-training weights are frozen, and only the joint encoder and decoder parameters are updated. This supervision mechanism improves the generalization accuracy of the model across power scenarios such as substations, transmission lines, and power distribution equipment, and avoids task overfitting.

[0126] In this embodiment of the invention, conditional decoding dynamically generates dual-type views based on preference vectors and simultaneously parses semantic information to achieve collaborative output of visual output and intelligent analysis results. Specifically, it includes preference vector parsing, dual-view generation, semantic parsing, and standardized output. Preference vector parsing and configuration include "view type (0 for human-friendly view, 1 for machine analysis view) + output precision ([0.3, 1.0])".

[0127] Dual-view generation:

[0128] Human-friendly view (Type 0): After decompression, the structure and texture features are inverted, and after color conversion (YUV→RGB) and resolution restoration (bilinear interpolation), a clear RG image is output;

[0129] Machine Analysis View (Type 1): Enhances contrast (gamma correction) and edges (Laplacian operator) in key areas, generates device masks and fault heatmaps (red / yellow / blue risk markers).

[0130] Semantic parsing, equipment category: matching the power terminology library with CLIP semantic vectors, and taking the highest similarity result (confidence ≥ 0.8); defect location: calculating the response value with a sliding window, and outputting the coordinates after non-maximum suppression (error ≤ 5 pixels).

[0131] The results are encapsulated in JSON format (including device type, status, defect coordinates, etc.) and pushed to the scheduling system via the MQTT protocol, linking with the work order system: high-risk cases are automatically generated into work orders, medium-risk cases are pending review, logs are kept for 3 months, and data anomalies are retried or marked for manual review.

[0132] Example 2

[0133] Please see Figure 3 , Figure 3 This is a schematic structural block diagram of an electronic device provided in an embodiment of this application.

[0134] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules. The processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used for signaling or data communication with other node devices.

[0135] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0136] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0137] It is understood that the structure shown in the figure is for illustrative purposes only. A multi-level visual coding and intelligent analysis method based on CLIP semantic supervision may include more or fewer components than shown in the figure, or have a different configuration. The components shown in the figure can be implemented in hardware, software, or a combination thereof.

[0138] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0139] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0140] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0142] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multi-level visual coding and intelligent analysis method based on CLIP semantic supervision, characterized in that, Includes the following steps: S1. Collect power scene videos, and generate preliminary structural features, texture features and semantic features by encoding the videos and retaining key image elements. S2. Input the preliminary structural features, texture features and semantic features into the joint encoder, and after being processed in layers by the multi-channel compression framework, they are fused into a unified feature representation, and the compressed feature data adapted to the transmission bandwidth and intelligent analysis requirements of the power scenario is output. S3. Using the pre-trained CLIP model as the source of semantic supervision, semantic consistency optimization is achieved by performing semantic supervision training on compressed feature data. S4. Perform conditional decoding on the optimized compressed feature data, dynamically generate corresponding views based on the preference vector, synchronously parse semantic information, and output intelligent analysis results containing power equipment status labels and defect area coordinates.

2. The multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in claim 1, characterized in that, Step S1 includes: Using real-time video streams of power scenarios as input, the system performs three levels of processing: salient region identification, region acquisition and encoding, and preliminary feature generation. The determination of salient regions adopts a two-dimensional determination mechanism of motion change rate and structural difference, and compares the current video frame with the previous frame frame by frame to select the regions that need to be collected and encoded. The calculation process for the rate of change of motion includes: ， In the formula, , represents a vector For vectors The norm; The calculation process for the structural difference includes: ， In the formula, It is a structural similarity index. For the current frame, For the previous frame, This represents the average local brightness. For local contrast, For local covariance, The stability coefficient; The preliminary feature generation includes the simultaneous generation of three types of preliminary features: structural features, texture features, and semantic features, while retaining key image elements of the device. The structural features are described above, and the gradient is calculated using the Sobel operator to obtain the gradient magnitude and direction. The calculation formula includes: ， ， In the formula, For gradient magnitude, For horizontal gradient, For vertical gradient, The gradient direction; The texture features are extracted using a Gabor filter kernel to obtain the surface texture and material features of the device. The calculation formula includes: ， ， ， In the formula, ( () represents the original image pixel coordinates in the Cartesian coordinate system. To control the carrier wavelength of the filtering frequency, Aspect ratio, Let be the standard deviation of the Gaussian envelope. , () represents the image pixel coordinates adjusted to the Cartesian coordinate system; The semantic features are input into the normalized salient region image of the CLIP visual encoder with frozen weights. After 12 layers of Transformer encoding, the mean pooling result of the last layer output is taken as the 512-dimensional semantic vector.

3. The multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in claim 2, characterized in that, Step S2 includes: Two-dimensional discrete wavelet transform (DWT) is used to perform multi-scale decomposition of structural features into low-frequency and high-frequency coefficients. The low-frequency coefficients containing the main contour information are retained, and threshold quantization is performed on the high-frequency coefficients to remove redundant details and output the low-frequency coefficients of structural features. The texture direction and frequency features are extracted by Gabor filtering. K-means clustering is used to group the texture feature vectors. The center vector of each cluster is retained as a representative texture feature. Duplicate texture data is reduced and the texture cluster center vector is output. Construct a semantic vector matrix based on the 512-dimensional semantic vector generated in step S1. By calculating the covariance matrix And solve for the covariance matrix. The eigenvalues ​​and corresponding eigenvectors are used to perform PCA dimensionality reduction on the semantic vector matrix X, outputting a semantic embedding vector. The calculation process includes: ， ， In the formula, Let covariance matrix be the variance matrix. Let X be the number of pixel blocks within the salient region, and let X be the semantic vector matrix. Let X be the mean vector. The semantic vector after dimensionality reduction. The projection matrix constructed from the eigenvectors; The structural feature low-frequency coefficients, texture clustering center vectors, and semantic embedding vectors are concatenated and mapped to a unified feature representation through a fully connected layer, outputting compressed feature data that adapts to the transmission bandwidth and intelligent analysis requirements of power scenarios.

4. The multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in claim 1, characterized in that, In step S3, the semantic consistency optimization includes: global scale alignment, local scale alignment, and instance scale alignment; The global scale alignment inputs the original image feature data and compressed feature data into the CLIP semantically supervised visual encoder, extracts the global semantic embedding vector, calculates the cosine similarity between the two and uses it as the global supervision loss. The local scale alignment divides the original image feature data and compressed feature data into 16×16 local blocks, extracts the CLIP semantic vector of each local block, and calculates the mean square error of the corresponding local block semantic vector as the local supervision loss. The instance scale alignment performs masking annotation on power equipment instances, extracts CLIP semantic vectors of corresponding instances from the original image feature data and compressed feature data, and calculates Euclidean distance as instance supervision loss.

5. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the processor executes the one or more programs, it implements a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a multi-level visual coding and intelligent analysis method based on CLIP semantic supervision as described in any one of claims 1-4.

Images