Scene adaptive video compression method and system based on natural language guidance

Through natural language guidance and in-depth analysis, combined with multi-directional adaptive wavelet transformation and hexagonal block matching algorithm, intelligent video compression in power inspection scenarios is achieved, solving the problem that traditional methods cannot adapt to the characteristics of different scenarios, and improving Compression efficiency and quality.

CN118972590AActive Publication Date: 2024-11-15ZHONGKE FANGCUN ZHIWEI (NANJING) TECH CO LTD

Patent Information

Application Number
CN202411434942.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-15
Publication Date
2024-11-15
Estimated Expiration
2044-10-15

AI Technical Summary

Technical Problem

Traditional video compression methods cannot flexibly adjust the compression ratio in power inspection scenarios, resulting in reduced visual quality and distortion of details, unable to fully balance compression efficiency and image quality, unable to adapt to the characteristics of different inspection scenarios and goals, resulting in quality loss and inefficiency. compression.

Method used

Adaptive video compression method based on natural language guidance is adopted to obtain video data and scene-related natural language text, generate enhanced scene description text, combine in-depth analysis and large language model, extract local and global features, and use six sides The block matching algorithm and the improved watershed algorithm are adaptively compressed, and further compressed using context adaptive binary arithmetic coding.

Benefits of technology

It realizes intelligent video compression in different scenarios, ensures the integrity of important information, and reduces the amount of data, improves video compression efficiency and quality, ensures image quality in key areas, is highly adaptable, and is suitable for complex scenarios such as power inspection. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972590A_ABST
    Figure CN118972590A_ABST
Patent Text Reader

Abstract

The invention discloses a scene adaptive video compression method and system based on natural language guidance, and the method comprises the steps: obtaining an original video data stream, and analyzing the original video data stream into a continuous video frame sequence; acquiring a natural language guide text related to the scene, determining the type of the current scene, and generating an enhanced scene description text; performing deep analysis by using a scene classification system to generate a scene feature vector; combining with a pre-trained large language model to generate an optimization strategy; extracting local and global features, and generating a comprehensive feature map; performing motion estimation to generate a motion vector and residual error information; fusing to generate a composite feature map; generating a mask by using a watershed algorithm; performing adaptive compression on the composite feature map, and outputting feature data; and further data compression is carried out by adopting context adaptive binary arithmetic coding, and a compressed data bit stream is output. According to the invention, the video compression efficiency and the quality of the compressed video are improved, and intelligent video compression in different scenes is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of video compression, and in particular to a scene-adaptive video compression method and system based on natural language guidance. Background Art

[0002] With the widespread application of digital video content, video compression technology has become increasingly important. Among them, power inspection is a scenario that requires frequent collection and transmission of video data. Power inspection usually involves monitoring and maintenance of power equipment and transmission lines to ensure the safety and stable operation of the power grid. In this process, efficient transmission and storage of video data are essential for accurate identification of problems and quick decision-making. Video compression aims to achieve efficient video transmission and storage by reducing the amount of data, reducing transmission bandwidth requirements and storage costs while maintaining acceptable visual quality. However, traditional video compression methods have some challenges, such as fixed compression algorithms and parameters that cannot adapt to the characteristics of different scenarios and content, resulting in quality loss or inefficient compression in some cases.

[0003] Traditional video compression methods have some challenges in power inspection scenarios. First, the compression ratio is an important indicator that determines the degree of compression of video data. A higher compression ratio can reduce data transmission and storage costs, but may lead to a decrease in visual quality and increased detail distortion, affecting the accurate identification of power inspection problems. Because traditional methods usually use a fixed compression ratio, they cannot be flexibly adjusted according to the characteristics of specific scenarios, and cannot fully balance compression efficiency and image quality. At the same time, power inspection involves different scenarios, such as high-altitude towers, substations, transmission lines, etc. Based on fixed compression algorithms and parameters, it is impossible to fully adapt to the characteristics of different inspection scenarios and inspection targets, which will cause quality loss and inefficient compression in some cases. Summary of the invention

[0004] The purpose of the invention is to provide a scene-adaptive video compression method and system based on natural language guidance to solve the above-mentioned problems existing in the prior art.

[0005] The technical solution is a scene-adaptive video compression method based on natural language guidance, comprising the following steps:

[0006] S1. Obtain an original video data stream, parse the original video data stream into a continuous sequence of video frames; obtain a natural language guide text related to the scene, construct a text feature vector, and calculate a scene relevance score based on the text feature vector and a predefined scene type template; determine the type of the current scene based on the scene relevance score; combine the type of the current scene with the natural language guide text to generate an enhanced scene description text; wherein the original video data stream includes the appearance status of the high-voltage power line, the surrounding environment conditions, and possible fault point features;

[0007] S2. Based on the enhanced scene description text, use the rule-based scene classification system to perform in-depth analysis and generate a scene feature vector; combine the scene feature vector with the pre-trained large language model to generate an optimization strategy for the current scene; based on the video frame sequence and the optimization strategy, use a multi-scale analysis method based on wavelet transform to extract local and global features; based on the local and global features, reconstruct high-resolution features and low-resolution features; combine high-resolution features and low-resolution features to generate a comprehensive feature map;

[0008] S3, based on the comprehensive feature map and the video frame sequence, use the hexagonal block matching algorithm to perform motion estimation, generate motion vectors and residual information; encode and compress the motion vectors and residual information, and output compressed motion information data; based on the comprehensive feature map and the compressed motion information data, perform feature fusion to generate a composite feature map;

[0009] S4, based on the composite feature map and the optimization strategy, using the improved watershed algorithm to generate foreground and background masks; based on the foreground and background masks, adaptively compressing the composite feature map, and outputting compressed feature data;

[0010] S5. Based on the compressed feature data, context-adaptive binary arithmetic coding is used to perform further data compression to output a final compressed data bit stream; based on the final compressed data bit stream, an error recovery mechanism and robustness enhancement measures are used to output a compressed data packet with error recovery capabilities.

[0011] A scene-adaptive video compression system based on natural language guidance, including:

[0012] at least one processor; and,

[0013] a memory communicatively connected to at least one of the processors; wherein,

[0014] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the scene-adaptive video compression method based on natural language guidance.

[0015] Beneficial effect: The present invention proposes a scene-adaptive video compression method based on natural language guidance, which adopts different compression strategies for the foreground area and the background area, so that the compressed video can not only ensure the integrity of important information, but also effectively reduce the amount of data; at the same time, it also improves the efficiency of video compression and the quality of the compressed video, and realizes intelligent video compression in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 This is a flow chart of step S1 of the present invention.

[0018] Figure 3 This is a flow chart of step S2 of the present invention.

[0019] Figure 4 This is a flow chart of step S3 of the present invention.

[0020] Figure 5 This is a flow chart of step S4 of the present invention.

[0021] Figure 6 This is a flow chart of step S5 of the present invention.

[0022] Figure 7 The figure is a logic flow chart of an embodiment of the present invention.

[0023] Figure 8 4 is a network structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 As shown, the present application proposes a scene-adaptive video compression method based on natural language guidance, comprising the following steps:

[0025] S1. Obtain an original video data stream, parse the original video data stream into a continuous sequence of video frames; obtain a natural language guide text related to the scene, construct a text feature vector, and calculate a scene relevance score based on the text feature vector and a predefined scene type template; determine the type of the current scene based on the scene relevance score; combine the type of the current scene with the natural language guide text to generate an enhanced scene description text; wherein the original video data stream includes the appearance status of the high-voltage power line, the surrounding environment conditions, and possible fault point features;

[0026] S2. Based on the enhanced scene description text, use the rule-based scene classification system to perform in-depth analysis and generate a scene feature vector; combine the scene feature vector with the pre-trained large language model to generate an optimization strategy for the current scene; based on the video frame sequence and the optimization strategy, use a multi-scale analysis method based on wavelet transform to extract local and global features; based on the local and global features, reconstruct high-resolution features and low-resolution features; combine high-resolution features and low-resolution features to generate a comprehensive feature map;

[0027] S3, based on the comprehensive feature map and the video frame sequence, use the hexagonal block matching algorithm to perform motion estimation, generate motion vectors and residual information; encode and compress the motion vectors and residual information, and output compressed motion information data; based on the comprehensive feature map and the compressed motion information data, perform feature fusion to generate a composite feature map;

[0028] S4, based on the composite feature map and the optimization strategy, using the improved watershed algorithm to generate foreground and background masks; based on the foreground and background masks, adaptively compressing the composite feature map, and outputting compressed feature data;

[0029] S5. Based on the compressed feature data, context-adaptive binary arithmetic coding is used to perform further data compression to output a final compressed data bit stream; based on the final compressed data bit stream, an error recovery mechanism and robustness enhancement measures are used to output a compressed data packet with error recovery capabilities.

[0030] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0031] S11. Obtaining an original video data stream through a power inspection device, preprocessing the original video data stream, removing damaged or incomplete data packets, and obtaining a preprocessed video data stream; extracting metadata information based on the preprocessed video data stream, including resolution, frame rate, and encoding format; parsing the original video data stream into a continuous video frame sequence based on the metadata information;

[0032] S12, obtaining natural language guidance text related to the current scene from a user input or a preset scene description library, preprocessing the natural language guidance text, including removing special characters, unifying upper and lower cases and segmenting words, to obtain a preprocessed natural language guidance text; using a pre-trained natural language processing model to perform part-of-speech tagging and named entity recognition on the preprocessed natural language guidance text to obtain a tagged text; extracting keywords and phrases from the tagged text to construct a text feature vector; performing similarity calculation based on the text feature vector and a predefined scene type template to obtain a similarity calculation result; based on the similarity calculation result, preliminarily determining the type of the current scene to obtain a preliminarily determined scene type;

[0033] S13, based on the preprocessed natural language guide text, perform deep semantic analysis to extract core semantic features; match the core semantic features with entries in a pre-built scene feature dictionary to calculate a matching score; based on the matching score and a preset weight system, calculate a relevance score for each scene type in the preliminarily determined scene types, and select the scene type with the highest relevance score as the type of the current scene finally determined;

[0034] S14. Based on the final determined type of the current scene, a corresponding template is selected from a preset scene description template library to obtain a selected scene description template; the preprocessed natural language guidance text is merged with the selected scene description template to generate an enhanced scene description text.

[0035] In one embodiment of the present application, an original video data stream is obtained, which comes from a power inspection device, including a high-definition camera carried by a drone, a ground-mounted surveillance camera, or other mobile detection equipment. A natural language guide text related to the current scene is received, such as "This is a power inspection application scenario, focusing on poles and high-voltage lines, please compress the input video." The obtained original video data stream is parsed into a continuous sequence of video frames. Pixel information is extracted for each frame of video data and converted into a three-dimensional tensor form, containing information of three color channels: red, green, and blue. The natural language guide text is input into a pre-trained rule-based anti-scene classification system. The input text is semantically analyzed, keywords are extracted and matched with entries in the dictionary. The scene relevance score is calculated, and the type of the current scene is determined based on the score result, such as power inspection, video surveillance, or video conferencing. The determined scene type information is combined with the original natural language guide text to generate an enhanced scene description text. The processed video frame sequence data and the enhanced scene description text are output for use in subsequent steps. The rule-based anti-scene classification system includes a predefined scene feature dictionary and a corresponding weight system.

[0036] This embodiment enhances the understanding of the scene by introducing a rule-based scene classification system and semantic analysis, and improves the adaptability of the video compression process to different scenes. The predefined scene feature dictionary and weight system can accurately capture the key features of specific scenes such as power inspection and video surveillance, thereby providing more accurate guidance for the subsequent compression process. Not only does it improve the accuracy of scene recognition, but it also provides rich contextual information for the compression algorithm. In the power inspection scene, the system can accurately identify key elements such as high-voltage wires and substations, and adjust the compression strategy accordingly. For example, for the details of the power equipment, the system will use a lower compression ratio to retain important information; while for the background area, a higher compression ratio can be used. This intelligent scene adaptability improves compression efficiency while ensuring image quality in key areas. By closely combining natural language guided text with video data, the system can better understand the operator's intentions, such as instructions such as "pay attention to poles and high-voltage lines" can directly affect the compression strategy. This not only improves the pertinence of compression, but also enhances the operability and flexibility of the system. By efficiently parsing and preprocessing the original video data stream, such as converting video frames into three-dimensional tensor form, a more suitable input format is provided for the subsequent deep learning model, thereby improving the efficiency of feature extraction and analysis. This embodiment lays a solid foundation for the entire video compression process, and through intelligent scene understanding and data preprocessing, it improves the performance of the system in specific scenarios such as power inspection.

[0037] like Figure 3As shown, according to one aspect of the present application, step S2 is further:

[0038] S21. Based on the enhanced scene description text, use an improved word frequency-inverse document frequency algorithm to calculate the weight of each word; based on the calculated weight, select a corresponding scene feature vector from a preset weight system;

[0039] S22. Use the pre-trained Sentence-BERT model to encode the enhanced scene description text to generate a semantic vector; concatenate the semantic vector with the scene feature vector to form an enhanced semantic representation; use a multi-layer perceptron to reduce the dimension of the enhanced semantic representation to generate a compressed semantic vector; apply the attention mechanism to calculate the importance weights of each dimension in the compressed semantic vector; based on the importance weights, perform weighted summation on the compressed semantic vector to obtain the final scene semantic features;

[0040] S23, inputting the scene semantic features into the pre-trained large language model to generate a preliminary compression strategy description for the current scene; matching the preliminary compression strategy description with a predefined compression rule library to obtain a matching result; based on the matching result, selecting a compression parameter combination from the compression rule library; based on the compression parameter combination and the preliminary compression strategy description, constructing an optimization strategy including initialization parameters;

[0041] S24, applying an improved discrete wavelet transform to decompose each frame image of the video frame sequence to obtain multi-scale sub-band coefficients, including low-frequency sub-band coefficients and high-frequency sub-band coefficients; applying a high-resolution encoder to the low-frequency sub-band coefficients to generate a global feature vector; applying a low-resolution encoder to the high-frequency sub-band coefficients to generate a local texture feature vector; using an adaptive threshold method to perform soft threshold processing on the multi-scale sub-band coefficients to obtain denoised coefficients; uniformly quantizing the denoised coefficients to obtain discretized coefficient values; encoding based on the discretized coefficient values ​​using run-length coding to obtain a compressed coefficient sequence;

[0042] S25. Based on the global feature vector, local texture feature vector and compressed coefficient sequence, the improved inverse discrete wavelet transform is used to reconstruct the high-resolution image and the low-resolution image; a convolutional neural network is applied to the high-resolution image to obtain high-resolution features; a lightweight convolutional neural network is applied to the low-resolution image to obtain low-resolution features; an attention mechanism is used to calculate the weight coefficients of the high-resolution features and the low-resolution features; based on the weight coefficients, the high-resolution features and the low-resolution features are weightedly summed to obtain a comprehensive feature map.

[0043] According to one aspect of the present application, the comprehensive feature map of step S25 is optimized, specifically:

[0044] S25a, applying a non-local means algorithm to process the comprehensive feature map to obtain a denoised feature map;

[0045] S25b. Based on the denoised feature map, adaptive histogram equalization is used to adjust the contrast to obtain an optimized comprehensive feature map.

[0046] According to one aspect of the present application, step S2 further includes:

[0047] S261, based on the parameters in the optimization strategy, adaptively quantize the comprehensive feature map to obtain a quantized feature map; use a content-based adaptive filter to process the quantized feature map to obtain a smoothed feature map;

[0048] S262, based on the smoothed feature map, using a local quality assessment method based on structural similarity to generate a quality map of the feature map; based on the quality map, identifying key areas and generating a key area mask;

[0049] S263, based on the key area mask, using a local contrast enhancement algorithm to obtain enhanced key area features; based on the enhanced key area features, using an image segmentation-based region growing algorithm to obtain optimized edge and contour information;

[0050] S264. Based on the optimized edge and contour information, a sparse representation method is used to obtain a sparse coefficient matrix; the sparse coefficient matrix is ​​reduced in dimension using the principal component analysis method to obtain a final low-dimensional feature representation.

[0051] According to another aspect of the present application, a dot product operation is performed on the text feature and the scene weight vector to obtain a scene relevance score; based on the relevance score, the most matching scene type is determined.

[0052] In one embodiment of the present application, scene feature extraction is performed based on an improved term frequency-inverse document frequency (TF-IDF) algorithm, specifically: receiving an enhanced scene description text, and using a natural language processing toolkit to segment the text. Stop words and punctuation marks are removed. Words are morphologically restored, and different forms of words are converted into their basic forms to obtain a preprocessed word list. A subset related to the current scene is extracted from a predefined scene feature dictionary, and the preprocessed word list is matched with a scene-specific sub-dictionary to generate a matched word list and an unmatched word list.

[0053] For the words in the matching word list, calculate their word frequency (TF) in the current text. For the words in the unmatched word list, use the sliding window technique to calculate their co-occurrence frequency with adjacent words. Use the co-occurrence frequency as the dynamic word frequency of the unmatched words. Combine the word frequency information of the matching words and the unmatched words to form a dynamic word frequency table. Calculate the document frequency of each word in the scene-specific dictionary. Use the sigmoid function to normalize the document frequency to obtain the normalized inverse document frequency (IDF). For words that do not appear in the scene-specific dictionary, use their IDF values ​​in the general corpus as the basis, and dynamically adjust them in combination with the characteristics of the current text to obtain the output adaptive IDF value table.

[0054] Calculate the basic TF-IDF value for each word, using a sliding window technique to consider the contextual information of each word. Calculate the semantic similarity of the words in the window, using pre-trained word vector models such as Word2Vec or GloVe. Adjust the basic TF-IDF value based on the semantic similarity, increase the weight of semantically related words, and output a table of context-aware TF-IDF values. Use principal component analysis (PCA) to reduce the dimension of the TF-IDF value and retain the main features. Apply L2 normalization to standardize the reduced features to generate the final scene feature vector. Apply sparse coding techniques such as the LASSO algorithm to the feature vector according to the parameters in the optimization strategy, and calculate the importance score of each dimension in the feature vector. Based on the importance score, select the top-K most important feature dimensions. Apply nonlinear transformations such as the hyperbolic tangent function to the selected feature dimensions to enhance the expressiveness of the features. Reorganize and compress the feature vector to generate the final optimized scene feature vector.

[0055] Perform dot product operation on text features and scene feature vectors to obtain scene relevance score. Based on the relevance score, determine the best matching scene type. Use pre-trained Sentence-BERT model (such as BGE-micro-v2) to encode the enhanced scene description text and generate a 768-dimensional semantic vector. Concatenate the semantic vector with the scene feature vector to form an enhanced semantic representation. Use multi-layer perceptron (MLP) to reduce the dimension of the enhanced semantic representation and generate a 256-dimensional compressed semantic vector. Apply the attention mechanism to calculate the importance weights of each dimension in the compressed semantic vector. According to the calculated weights, perform weighted summation on the compressed semantic vector to obtain the final scene semantic features.

[0056] Input the scene semantic features into a pre-trained large language model (such as GPT-3 or BERT). Use the model to generate a preliminary compression strategy description for the current scene. Match the generated strategy description with the predefined compression rule base. Based on the matching results, select the most suitable compression parameter combination from the rule base. Use the selected parameter combination to initialize the various parameters of the compression algorithm (such as quantization level, motion estimation accuracy, etc.). Generate an optimization strategy data structure containing the initialization parameters to guide the subsequent video compression process.

[0057] Divide each frame image in the video frame sequence data into overlapping blocks of fixed size (such as 16x16 pixels, overlapping 4 pixels). Perform symmetrical boundary extension on each image block to reduce edge effects. Output the set of expanded image blocks. Based on the set of expanded image blocks, apply the Sobel operator to each image block to calculate the horizontal and vertical gradients. Use the gradient information to build a block-level direction histogram. Apply the main direction detection algorithm to determine the main direction of each block (0°, 45°, 90°, 135°). Output the main direction information of each block.

[0058] According to the main direction of each block, the best matching basis function is selected from the predefined directional Daubechies wavelet basis function library. For non-main directions, the standard Daubechies wavelet basis function is selected, and a block-level hybrid basis function set is constructed, and an adaptive basis function set for each block is output. A 3-level multiscale decomposition is applied to each image block. In each level of decomposition, the corresponding directional basis function is used for transformation according to the main direction of the block. For non-main directions, a standard 1D discrete wavelet transform (DWT) is used. Multi-scale, multi-directional subband coefficients containing low-frequency and high-frequency information are generated.

[0059] Adaptive thresholding technique is applied to soft threshold processing of high-frequency subband coefficients. Local variance estimation is used to adjust the threshold to better preserve edge and texture information. Non-local mean filtering is applied to low-frequency subband coefficients to suppress noise while retaining structural information, and the optimized subband coefficients are output. The subband coefficients of each scale are directionally merged to generate a comprehensive feature map. Principal component analysis (PCA) is used to reduce the dimension of low-frequency subband coefficients and extract global features. Local binary pattern (LBP) operators are applied to high-frequency subband coefficients (horizontal, vertical, and diagonal) to extract local texture features. The global features and local texture features are concatenated to form the final feature vector.

[0060] Use vector quantization technology to preliminarily compress the feature vector. Apply run-length encoding (RLE) to encode the quantized feature vector. Use Huffman coding to further compress the RLE result. Generate a compressed feature data stream and the corresponding encoding dictionary.

[0061] Based on the multi-scale wavelet coefficients, global feature vectors, local texture feature vectors and compressed coefficient sequences, the improved inverse discrete wavelet transform (IDWT) is used to reconstruct high-resolution and low-resolution images. A convolutional neural network (CNN) is applied to the reconstructed high-resolution image to obtain a high-level semantic feature map. A lightweight CNN is applied to the reconstructed low-resolution image to obtain texture and edge feature maps. The weight coefficients of high-resolution and low-resolution features are calculated using the attention mechanism. According to the calculated weight coefficients, the high-resolution and low-resolution feature maps are weighted summed to obtain a fused feature map. The non-local means algorithm is applied to the fused feature map to obtain a denoised feature map. The denoised feature map is processed using adaptive histogram equalization (CLAHE) to obtain a contrast-adjusted feature map, and the final fused enhanced feature map is output.

[0062] According to the parameters in the optimization strategy, adaptive quantization is applied to the feature map to obtain the quantized feature map. The quantized feature map is processed using a content-based adaptive filter to obtain a smoothed feature map. A local quality assessment method based on structural similarity (SSIM) is applied to generate a quality map of the feature map. According to the quality map, the key areas in the feature map are identified and a key area mask is generated. A local contrast enhancement algorithm is applied to the identified key areas to obtain enhanced key area features. The feature map is processed using a region growing algorithm based on image segmentation to obtain optimized edge and contour information. The feature map is processed using a sparse representation method to obtain a sparse coefficient matrix. The sparse coefficient matrix is ​​reduced in dimensionality using principal component analysis (PCA) to obtain the final low-dimensional feature representation.

[0063] This embodiment enhances the system's ability to understand and utilize scene semantic information by introducing an improved TF-IDF algorithm and a context-aware feature extraction method, thereby providing more accurate guidance for the subsequent video compression process, and showing advantages in specific scenarios such as power inspection. Dynamic word frequency calculation takes into account the importance of words in the current text and the relationship between adjacent words, and can more accurately capture the importance changes of key terms such as "high-voltage line" and "substation" in different inspection scenarios. Adaptive inverse document frequency calculation better balances the weights of common words and professional terms by introducing sigmoid function normalization and dynamic adjustment mechanism, enabling the system to accurately identify specific descriptions of power equipment. Context-aware TF-IDF calculation enhances the system's ability to understand complex power inspection instructions by considering the semantic relevance of words, such as "check the joint status of the transmission line" can be correctly parsed and affect subsequent compression strategies. The feature vector generation and optimization process, especially the sparse coding using PCA dimensionality reduction and LASSO algorithm, not only reduces the data dimension, but also retains the most critical scene features, such as the type, location and status information of power equipment. These optimizations enable the system to quickly process large amounts of inspection video data with limited computing resources. The application of nonlinear transformations enhances the expressive power of features, enabling the system to better distinguish subtle scene differences, such as the features of normally operating equipment and potentially faulty equipment. This highly optimized scene feature extraction method improves the pertinence and efficiency of the subsequent video compression process. For example, when the joint area of ​​a high-voltage line is identified, the system can automatically adjust to a low compression ratio mode to retain possible fault details; while for large areas of uniform backgrounds, such as the sky or the ground, a higher compression ratio can be used. This intelligent compression strategy not only significantly reduces the amount of data transmitted, but also ensures the image quality of key areas, providing more reliable visual information for power inspectors. This embodiment improves the intelligence level and practicality of the system in professional scenarios such as power inspections.

[0064] By introducing improved multi-directional adaptive wavelet transform and multi-scale analysis methods, the system's ability to capture complex textures and directional structures in power inspection videos is enhanced, thereby achieving more efficient and accurate video compression, which is particularly suitable for processing complex structures and details of power facilities. Frame segmentation and boundary extension technology reduces block effects and improves compression quality, especially when dealing with linear structures such as high-voltage wires and towers. Directional analysis and adaptive basis function selection can accurately capture the different directional features of power equipment, such as vertical poles, inclined guy wires, and horizontal crossarms, ensuring that these key structures can still be clearly identified after compression. The application of multi-scale directional transform enables the system to simultaneously retain large-scale structures (such as the overall tower outline) and small-scale details (such as tiny defects on the surface of insulators), which is crucial for a comprehensive assessment of the condition of power facilities. The sub-band coefficient optimization process, especially the application of adaptive threshold and non-local mean filtering, effectively suppresses noise while retaining important details, improves the signal-to-noise ratio of compressed videos, and helps to more accurately identify potential equipment failures. In the coefficient reorganization and feature extraction stage, advanced technologies such as PCA and LBP are used to effectively extract global and local features, allowing the system to focus on large-scale power grid layout and local equipment details at the same time. The vector quantization and improved entropy coding technology used in the feature compression and encoding stage not only greatly reduces the amount of data, but also maintains the integrity of key information, so that the compressed video can still transmit enough details for analysis under limited bandwidth. This embodiment is conducive to the rapid transmission of large amounts of inspection video data in remote monitoring, while ensuring that operators can clearly observe the status of key equipment. For example, when observing transmission lines, the system can simultaneously maintain subtle changes in the surface of the conductor (such as possible signs of corrosion) and the curvature information of the entire line, which is crucial for timely detection of potential problems. This embodiment improves the compression efficiency and quality of power inspection videos, and provides strong technical support for the safe operation and preventive maintenance of power systems.

[0065] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0066] S31, dividing the current frame and the reference frame of the video frame sequence into overlapping hexagonal blocks; based on the comprehensive feature map, for each hexagonal block in the current frame, performing block matching within the search window of the reference frame to obtain a predetermined number of candidate matching positions; calculating the matching cost of each candidate matching position, and selecting the position with the smallest matching cost as the best matching position; calculating the displacement between the current block and the best matching position to obtain an initial motion vector; using a half-pixel precision interpolation method to refine the initial motion vector to obtain an optimized motion vector;

[0067] S32, based on the optimized motion vector, perform motion compensation on the reference frame to generate a predicted frame; calculate the difference between the predicted frame and the current frame to obtain residual data; apply discrete cosine transform to the residual data to obtain frequency domain residual coefficients; use an improved adaptive quantization method to quantize the frequency domain residual coefficients to obtain quantized residual coefficients;

[0068] S33, based on the optimized motion vector, differential encoding is performed to obtain a differential motion vector; Huffman coding is used to entropy encode the differential motion vector to obtain compressed motion vector data; run-length coding is applied to the quantized residual coefficient to obtain compressed residual data; context-adaptive binary arithmetic coding is used to further compress the compressed residual data to obtain final compressed residual data; the compressed motion vector data and the final compressed residual data are combined to generate final motion information data;

[0069] S34, up-sample the scene semantic features to make their dimensions match the comprehensive feature map; decode the motion information data into a motion feature map, and use a linear interpolation method to adjust the size of the motion feature map to make it match the comprehensive feature map. Figure 1 Output the aligned scene semantic features, comprehensive feature maps and motion feature maps;

[0070] S35. Based on the aligned scene semantic features, comprehensive feature maps and motion feature maps, the attention mechanism is used to calculate the feature weight coefficients; based on the feature weight coefficients, weighted summation is performed to obtain preliminary fusion features; the preliminary fusion features are processed using a 3D convolutional neural network to obtain processed features; the aligned scene semantic features, comprehensive feature maps and motion feature maps are added to the processed features using residual connections to obtain the final composite feature map.

[0071] According to one aspect of the present application, the motion information data in step S33 is optimized, specifically:

[0072] S33a, based on the parameters in the optimization strategy, using a rate-distortion optimization algorithm, performing adaptive bit allocation on the motion information data to obtain a bit allocation result;

[0073] S33b, based on the bit allocation result, adopting the sliding window technology, re-encoding the motion information data to obtain optimized motion information data;

[0074] S33c, based on the optimized motion information data, perform bit stream packaging to obtain a final bit stream format, and generate corresponding decoding instructions.

[0075] In one embodiment of the present application, a low-dimensional feature representation and video frame sequence data are received, and a hexagonal grid covering the entire frame is constructed using a regular hexagon as a basic unit. The hexagon size is adjusted so that its side length is 4-8 pixels (dynamically adjustable according to the video resolution). Deformed hexagons are used at the grid boundaries to ensure complete coverage of the frame, and the hexagonal grid structure and the coordinate information of each hexagonal block are output. A Gabor filter group is applied to each hexagonal block to enhance the texture features. The gradient direction histogram of each hexagonal block is calculated. Using the main direction detection algorithm, the main direction (0°, 60°, 120°) of each block is determined, and the enhanced hexagonal block features and main direction information are output.

[0076] Based on the motion information of the previous frame, an initial search center is defined for each hexagonal block. According to the main direction of the block, a directional search window is constructed and extended along the main direction. For areas with intense motion, the search window is appropriately enlarged; for static areas, the search window is reduced, and the adaptive search window parameters for each hexagonal block are output. A three-level hierarchical matching strategy is implemented: 1) initial matching is performed at a coarse scale, using downsampled block features; 2) the matching results are refined at a medium scale; 3) the final matching is performed at the original scale. At each scale, a sliding window search is performed using a matching template in the shape of a hexagon. The matching cost is calculated, and the weighted sum of absolute differences (WSAD) is used as the similarity metric, and the best matching position at each scale is output.

[0077] Interpolate in six directions around the best matching position to generate candidate points with 1 / 4 pixel accuracy. Use parabola fitting method to find the best matching position with sub-pixel accuracy among these candidate points. Calculate the motion vector accurate to 1 / 4 pixel. Apply median filtering to perform preliminary smoothing on the motion vector field. Calculate the consistency index of the local motion vector. Identify abnormal blocks that are inconsistent with the motion of the six surrounding neighboring blocks. For abnormal blocks, use the motion information of the surrounding consistent blocks to perform interpolation correction and output a smooth and consistent motion vector field.

[0078] Based on a smooth and consistent motion vector field, calculate the motion complexity and texture complexity of each hexagonal block. For areas with complex motion and texture, split a hexagonal block into seven smaller hexagonal blocks. For areas with simple motion and texture, merge the seven small hexagonal blocks into a large hexagonal block. Repeat the above steps to get the optimized motion vector field. Output the final motion vector field and adaptive block size information.

[0079] According to the motion vector field, motion compensation is performed on the reference frame to generate a predicted frame. The difference between the current frame and the predicted frame is calculated to obtain the initial residual data. The residual data is divided into the same hexagonal block structure as in the above step, and the hexagonal residual block is output.

[0080] For each hexagonal residual block, the following operations are performed:

[0081] 1. Calculate the main direction of the pixels in the block (0°, 60°, 120°).

[0082] 2. Map the hexagonal blocks to the closest rectangular area according to the main direction.

[0083] 3. Apply directional discrete cosine transform (DCT) to the mapped rectangular area, and adjust the basis functions along the main direction.

[0084] 4. Map the DCT coefficients back to the hexagonal structure. Output the DCT coefficients of the hexagonal structure.

[0085] Analyze the statistical characteristics of DCT coefficients, including coefficient distribution and energy concentration. Generate an initial quantization matrix based on the analysis results and optimization strategy. Consider the visual characteristics of the human eye and perform perceptual weighting on the quantization matrix. Adaptively adjust the quantization matrix based on the local texture complexity to obtain an adaptive quantization matrix.

[0086] Implement improved dead zone quantization: use a narrower dead zone for low-frequency DCT coefficients to retain more details; use a wider dead zone for high-frequency DCT coefficients to increase sparsity; adaptively adjust the dead zone width based on coefficient amplitude and local activity, and output the DCT coefficients after preliminary quantization.

[0087] Perform the rate-distortion optimization (RDO) process: define multiple candidate quantization parameter sets; for each parameter set, calculate the quantized bit rate and distortion; use the Lagrange multiplier method to find the best rate-distortion balance point; apply the selected optimal quantization parameters, re-quantize the DCT coefficients, and output the quantized DCT coefficients after RDO optimization.

[0088] Dequantize the DCT coefficients to obtain the preliminary reconstructed residual data; analyze the distribution characteristics of the quantization noise; construct an adaptive deblocking filter to reduce the block effect; apply content-based sharpening filtering to enhance edges and textures; update the quantized DCT coefficients to make them consistent with the filtered residual. Output the final quantized DCT coefficients and the corresponding filtering parameters. Rearrange the quantized DCT coefficients from hexagons to rectangles to facilitate subsequent encoding; perform improved zigzag scanning, considering the special structure of hexagonal blocks; generate a cascaded one-dimensional coefficient sequence; record the position information of non-zero coefficients, generate an importance map, and output the rearranged one-dimensional quantized coefficient sequence, importance map, and filtering parameters.

[0089] Differentially encode the motion vector field to obtain differential motion vectors. Entropy encode the differential motion vectors using Huffman coding to obtain compressed motion vector data. Apply run-length encoding (RLE) to the quantized residual coefficient sequence to obtain compressed residual data. Use context-adaptive binary arithmetic coding (CABAC) to further compress the RLE-encoded residual data. Combine the compressed motion vector data and residual data to generate the final motion information data stream. Add necessary header information and metadata to the data stream and output the compressed motion information data stream.

[0090] This embodiment introduces an improved adaptive hexagonal block matching algorithm to improve the accuracy and efficiency of motion estimation and compensation in the power inspection video, which is particularly suitable for processing complex motion patterns commonly seen in power facility inspections. The hexagonal grid generation technology provides a more flexible coverage method than traditional square blocks, and can better adapt to the irregular shapes of power equipment, such as curved high-voltage lines or complex tower structures. The feature enhancement and directional analysis steps, especially the application of Gabor filters, enhance the system's ability to capture texture features of power equipment, so that even in the case of changes in illumination or slight displacements, the movement of the equipment can be accurately tracked. The adaptive search window definition dynamically adjusts the search range based on the motion information of the previous frame, which is particularly effective in processing the swing of wires caused by wind or the rapid change of perspective during drone inspections, improving the accuracy and computational efficiency of motion estimation. The hierarchical block matching strategy can simultaneously capture large-scale camera motion (such as the flight trajectory of a drone) and local subtle changes (such as the accumulation of stains on the surface of an insulator) through multi-scale search, which is crucial for a comprehensive assessment of the status of power facilities. The introduction of sub-pixel precision optimization technology enables the system to detect extremely small movements, such as slight deformation of wires caused by thermal expansion and contraction. This high-precision motion estimation is of great significance for preventive maintenance. The motion vector smoothing and anomaly detection steps effectively eliminate erroneous estimates caused by image noise or occlusion, and improve the consistency and reliability of the motion field, which is particularly useful when dealing with complex power facility environments (such as lines partially obscured by vegetation). Adaptive block size adjustment technology dynamically adjusts the block size according to the motion and texture complexity of the area, and performs well in dealing with complex scenes such as substations that contain a variety of equipment. It can accurately capture the movement of small components and effectively handle the overall changes of large structures. This highly optimized motion estimation method not only improves the accuracy of subsequent inter-frame predictions, thereby improving compression efficiency, but also provides a reliable basis for anomaly detection. For example, the system can accurately identify equipment jitters that do not match normal motion patterns, which may indicate potential mechanical problems. At the same time, accurate motion compensation allows the compressed video to maintain clarity and coherence in key areas at low bit rates, which is critical for remote real-time monitoring and subsequent automated analysis.

[0091] Through the feature fusion method, scene semantics, low-dimensional visual features and motion information are effectively integrated, which improves the intelligence level and adaptability of the system in power inspection video compression. This multimodal feature fusion technology is particularly suitable for handling complex and changeable scenes in power inspection. The feature alignment step ensures the consistency of information from different sources in spatial and temporal dimensions, which is particularly important for accurately parsing facilities such as transmission lines that span a large area. The application of upsampling and linear interpolation techniques enables the system to seamlessly integrate information at different resolutions and scales, so that large-scale structures (such as the overall outline of power towers) can be retained during the compression process, while small-scale details (such as tiny defects on the surface of insulators) can be retained. The application of attention mechanism in feature fusion enables the system to dynamically adjust the importance of different types of information, which performs well in handling complex power facility inspection scenes. For example, when inspecting high-voltage lines, the system can automatically increase the weight of motion features to accurately capture the slight swing of the wire; while when inspecting stationary transformers, it can pay more attention to semantic and visual features to capture possible abnormal appearances. The introduction of 3D convolutional neural networks enables the system to effectively capture spatiotemporal correlations, which is particularly useful for analyzing timing-related fault phenomena such as insulator flashover. The application of residual connections ensures that the original feature information is not lost in multiple processing, which is essential for retaining key details of power equipment, such as early signs of corrosion or tiny mechanical deformations. This highly fused feature representation provides a rich and accurate information basis for the subsequent compression process, allowing the system to allocate bit resources more intelligently. For example, for parts identified as potential fault areas, the system can allocate more bits to retain details; while for normally operating equipment or background areas, a higher compression ratio can be used. This adaptive compression strategy not only greatly improves the overall compression efficiency, but also ensures the retention of key information, improving the practicality of compressed video in remote monitoring and fault diagnosis. The fused feature map also provides strong support for anomaly detection. Based on the fused features, the system can more accurately identify conditions that do not conform to normal patterns, such as abnormal heating of equipment (through visual features), abnormal vibration (through motion features), or equipment status that does not conform to the current inspection task (through semantic features). This comprehensive anomaly detection capability improves the efficiency and reliability of power inspection.

[0092] This embodiment improves the efficiency and quality of power inspection video compression through computer vision technology, providing more reliable and efficient visual data support for the safe operation and preventive maintenance of the power system; through multimodal feature fusion technology, the intelligence level of the power inspection video compression system is enhanced, providing a more comprehensive and reliable visual analysis basis for the safe operation, preventive maintenance and fault diagnosis of the power system.

[0093] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0094] S41, applying Gaussian filtering to the composite feature map to obtain a filtered image; based on the filtered image, calculating the gradient amplitude to obtain a gradient map; using morphological operations to process the gradient map to obtain a processed gradient map;

[0095] S42, based on the parameters in the optimization strategy, determine the threshold and seed point selection strategy of the watershed algorithm; apply the distance transform on the processed gradient map to obtain a distance map; use the H-minima transform to suppress the local minimum in the distance map to obtain a suppressed distance map; based on the suppressed distance map, select the seed point according to the threshold and seed point selection strategy; starting from the seed point, execute the improved watershed algorithm to obtain a preliminary segmentation result;

[0096] S43, based on the preliminary segmentation result, using a region merging algorithm to merge similar small regions to obtain a merged segmentation result; based on the merged segmentation result, using an edge-preserving filter to smooth the region boundary to obtain a smoothed segmentation result; based on preset scene semantic information, adjusting the division of foreground and background in the smoothed segmentation result to generate a binary mask;

[0097] S44. Based on the binary mask, the composite feature map is divided into a foreground area and a background area; fine-grained quantization is applied to the foreground area to obtain foreground area data; coarse-grained quantization is applied to the background area to obtain background area data; entropy coding is used for the foreground area data, and run-length coding is used for the background area data to generate a coded bit stream, i.e., compressed feature data.

[0098] In one embodiment of the present application, a Gaussian filter is applied to the composite feature map to reduce the influence of noise. The gradient amplitude of the filtered image is calculated to obtain a gradient map. The gradient map is processed using morphological operations (such as opening and closing operations) to highlight the main structure and obtain a processed gradient map.

[0099] According to the parameters in the optimization strategy, set the seed point selection threshold. Apply H-minima transform to suppress local minima in the gradient map. Calculate the local minimum area of ​​the gradient map. Calculate the area and average gradient value for each local minimum area. According to the area and gradient threshold, filter out the seed points that meet the conditions and output the selected seed point set. Assign a unique label to each seed point to obtain the marked seed point; create a priority queue and add the adjacent pixels of the seed point to the queue; calculate the gradient value of each adjacent pixel as the priority; initialize the watershed line set to empty, and output the initialized priority queue and the empty watershed line set.

[0100] Receives the marked seed point, the initialized priority queue, and an empty watershed line set. When the priority queue is not empty, perform the following operations:

[0101] 1. Take out the pixel P with the smallest gradient value from the queue.

[0102] 2. Check the labels of P’s neighboring pixels.

[0103] 3. If all labeled adjacent pixels have the same label, then P is labeled as the same label.

[0104] 4. If there are adjacent pixels with different labels, mark P as a watershed point.

[0105] 5. Add the unlabeled neighboring pixels of P to the priority queue. Update the watershed line set.

[0106] Output the updated label map and watershed line set.

[0107] Based on the label map and watershed line set, the average gradient value and area of ​​each region are calculated. The merge threshold is set, and the adjacent small regions are merged according to the area and gradient difference of the region, and the label map is updated; the watershed line is refined, and isolated short line segments are removed, and the optimized label map and refined watershed line are output.

[0108] In the narrow band area near the watershed line, perform the following operations:

[0109] 1. Calculate the local gradient direction.

[0110] 2. Fine-tune the watershed line along the gradient direction.

[0111] 3. Use spline interpolation to smooth the adjusted boundaries, update the label map to reflect the boundary adjustment results, and output the adjusted label map and optimized segmentation boundaries.

[0112] Map the scene semantic features to the segmented regions and calculate the semantic consistency score of each region. Identify semantically inconsistent regions and readjust the boundaries of inconsistent regions by combining semantic information and image gradients. Update the label map and segmentation boundaries and output a semantically consistent label map and the final segmentation boundaries.

[0113] Based on the semantically consistent label map and the final segmentation boundary, a region merging algorithm is applied to merge similar small regions. An edge-preserving filter is used to smooth the region boundaries. The foreground and background divisions are adjusted according to the scene semantic information. A binary mask is generated, in which the foreground region is marked as 1 and the background region is marked as 0. The composite feature map is divided into foreground and background regions according to the mask. Fine-grained quantization is applied to the foreground region to retain more details. Coarse-grained quantization is applied to the background region to reduce the amount of data. The quantization step size is adjusted using the parameters in the optimization strategy, and the quantized feature data is output. An efficient entropy coding method (such as arithmetic coding) is used for the foreground region data; a simple run-length coding is used for the background region data; and an encoded bitstream is generated, including foreground and background data.

[0114] This embodiment improves the accuracy and reliability of the system's foreground and background segmentation in power inspection videos by introducing an improved watershed algorithm and semantic consistency check, providing a solid foundation for subsequent adaptive compression, and is particularly suitable for processing complex equipment structures and changing environmental conditions in power inspection scenes. The applied Gaussian filtering and morphological operations effectively reduce the impact of noise and highlight the main structure, which is particularly important when processing slender structures such as transmission lines, ensuring that the line will not be broken during the segmentation process. The improved watershed algorithm improves the ability to capture the contours of complex power equipment, such as the multi-layer structure of a transformer or the grid structure of a tower, by introducing H-minima transformation and adaptive threshold selection. This precise segmentation provides accurate regional division for the subsequent compression process, enabling the system to adopt different compression strategies for important equipment components and background areas. The introduction of Dead Zone quantization, especially the use of Dead Zones of different widths for low-frequency and high-frequency DCT coefficients, effectively increases the compression ratio of the background area while retaining the key details of the power equipment, which is particularly important for transmitting a large amount of inspection video data under limited bandwidth. The rate-distortion optimized quantization process ensures that the clarity of key equipment is not significantly affected during the compression process by intelligently balancing bit rate and image quality, while also maximizing bandwidth utilization efficiency. The application of adaptive reconstruction loop filtering effectively reduces block effects and enhances edges and textures, which is particularly useful when observing stains on the surface of insulators or subtle corrosion on metal parts, improving the practicality of compressed videos in fault diagnosis. The semantic consistency check combines scene semantic information with image segmentation results to ensure that the segmentation results are consistent with the actual needs of power inspection. For example, the system can accurately identify and retain the complete structure of the transmission tower, even if part of the tower may be similar in color to the background; at the same time, it can correctly distinguish between actual power equipment and other objects that may appear in the video, such as flying birds or branches, avoiding unnecessary high-definition compression. This semantically guided segmentation method not only improves compression efficiency, but also enhances the system's ability to identify key areas, laying the foundation for subsequent automated analysis and fault detection. The boundary adjustment and optimization process, especially the use of spline interpolation for boundary smoothing, further improves the accuracy and visual quality of segmentation, which performs well when dealing with complex boundaries such as the junction of high-voltage lines and the sky. The adaptability and robustness of the entire segmentation process enable the system to effectively cope with various complex power inspection scenarios, from densely packed substation equipment to transmission lines in open areas, and obtain high-quality segmentation results. This precise foreground and background segmentation not only directly improves compression efficiency, but also provides reliable input for subsequent advanced analysis tasks such as target tracking and anomaly detection.

[0115] By introducing adaptive quantization and encoding technology, the system's efficiency and quality control capabilities in power inspection video compression are improved, making it suitable for processing complex and changeable scenes and image areas of different importance in power inspection. The adaptive quantization step performs differential processing on the composite feature map based on the foreground background mask, showing extremely high efficiency in processing power inspection videos. For foreground areas, such as key equipment such as high-voltage wires, transformers or insulators, the system uses fine-grained quantization to retain more detailed information. This ensures that tiny features that may indicate equipment failure or abnormality, such as fine cracks on the surface of insulators or slight rust on metal parts, are not lost during the compression process. In contrast, for background areas, such as the sky or distant vegetation, the system uses coarse-grained quantization, reducing the amount of data without affecting the transmission of key information. This differentiated quantization strategy not only improves the overall compression efficiency, but also ensures that the most critical visual information is retained in the video transmitted under limited bandwidth. The dynamic adjustment mechanism of the quantization step size is adjusted in real time according to the parameters in the optimization strategy, enabling the system to flexibly respond to different inspection task requirements and network conditions. For example, when inspecting key joints of transmission lines, the system can automatically reduce the quantization step size to improve image quality; while when inspecting a large range of normally operating lines, the step size can be appropriately increased to improve transmission efficiency. In the adaptive coding step, different coding strategies are used for the foreground and background areas to further optimize the compression performance. The use of efficient entropy coding methods, such as arithmetic coding, for the foreground area ensures high-fidelity transmission of key equipment information. This is particularly important in remote real-time monitoring, allowing operators to clearly observe subtle changes in the status of the equipment. Simple run-length coding is used for the background area, which greatly reduces the amount of data while still maintaining sufficient contextual information to help understand the overall scene. This differentiation of coding strategies not only improves compression efficiency, but also provides more valuable data input for subsequent intelligent analysis. For example, in the automatic fault detection algorithm, the system can give priority to high-quality encoded foreground areas to improve detection accuracy and efficiency. This adaptive compression method also improves the system's adaptability under different network conditions. When bandwidth is limited, the system can automatically increase the compression ratio of the background area to ensure timely transmission of key equipment information; when network conditions are good, it can provide more comprehensive scene information, which helps to make more in-depth analysis and decision-making. This intelligent bandwidth allocation not only optimizes the use of network resources, but also improves the reliability and efficiency of the entire power inspection system.

[0116] This embodiment improves the intelligence level and adaptability of the power inspection video compression system, and provides strong technical support for the safe monitoring, preventive maintenance and efficient operation of the power system; at the same time, it also improves the performance and adaptability of the power inspection video compression system, and provides more reliable and efficient visual data support for the real-time monitoring, fault diagnosis and preventive maintenance of the power system, which promotes the development of smart grids and the safe operation of power systems.

[0117] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0118] S51, constructing a context probability model based on the compressed feature data; encoding the compressed feature data using a context adaptive binary arithmetic coding algorithm based on the context probability model to generate a compressed data bit stream;

[0119] S52, encoding the compressed data bit stream using a Reed-Solomon code to generate a redundant check bit; combining the compressed data bit stream and the redundant check bit to form a data block with error correction capability;

[0120] S53, dividing the data blocks with error correction capability into data packets of a predetermined size, calculating the checksum of the cyclic redundancy check for each data packet; adding the checksum to the header of each data packet, and assembling them into compressed data packets with error recovery capability.

[0121] In one embodiment of the present application, the compressed feature data is parsed into a sequence of syntax elements, such as motion vectors, residual coefficients, etc. The frequency and distribution of each type of syntax element are calculated to generate a preliminary data statistics report. Three levels of context are defined:

[0122] Pixel-level context: consider the encoded pixels around the current pixel;

[0123] Block-level context: utilizing information from neighboring coded blocks;

[0124] Frame-level context: leverages the encoding information of previous frames.

[0125] Build a context template for each level and output a multi-level context template.

[0126] Based on the multi-level context templates and data statistics report, construct the initial conditional probability table (CPT) for each grammatical element type. Use the frequency information in the data statistics report to initialize the CPT. Apply Laplace smoothing to avoid zero probability problems and output the set of initialized conditional probability tables. Use small batches of training data (sampled from the current data) and apply online learning algorithms such as stochastic gradient descent to optimize the weight of each context. Calculate the log-likelihood ratio of each level of context as the basis for weight update and output the learned context weights. Construct a sliding window mechanism to continuously observe the most recently encoded data; dynamically update the conditional probability table based on the observed data; apply an exponential decay factor to balance the impact of historical data and new data; use the learned weights to fuse the probability estimates of the multi-level context and output the updated adaptive probability model.

[0127] According to the parameters in the optimization strategy, set the target range of model complexity. Calculate the complexity of the current model (such as the number of parameters, entropy). If the complexity is too high, perform model pruning to remove unimportant context or merge similar probability distributions; if the complexity is too low, add new context combinations or refine existing probability distributions, and output the adjusted adaptive probability model. Using the retained validation dataset, calculate the cross entropy and compression ratio of the model on the validation set. Compare the performance of the current model with the previous version. If the performance improvement is not significant, roll back to the previous model version and generate a model performance report.

[0128] Encode the data using the CABAC algorithm, dynamically adjust the encoding strategy based on the contextual probability model, and generate a compressed binary bit stream as the final compressed data bit stream. Encode the data using the Reed-Solomon code to generate redundant check bits. Combine the original data and the check bits to form a data block with error correction capabilities. Divide the data block into fixed-size data packets, calculate the CRC (cyclic redundancy check) checksum for each data packet, add the checksum to the header of each data packet, and assemble the final data packet, including the original data, error correction code, and checksum. Output a compressed data packet with error recovery capabilities.

[0129] This embodiment improves the coding efficiency and adaptability of the system in power inspection video compression by introducing context-adaptive binary arithmetic coding (CABAC) and a dynamic probability model update mechanism, and is suitable for processing highly structured and dynamically changing visual content in power inspection videos. By parsing the compressed data stream into a sequence of syntax elements, such as motion vectors and residual coefficients, a refined basis is provided for subsequent probabilistic modeling. This fine-grained data analysis enables the system to capture the unique visual features and motion patterns of power equipment, such as the periodic swing of high-voltage lines or the regular texture of transformer heat sinks. The introduction of multi-level context definitions, including pixel-level, block-level, and frame-level contexts, enables the encoding process to fully utilize the spatiotemporal correlation of power inspection videos. For example, when encoding transmission lines, the system can use the high correlation of adjacent pixels to improve compression efficiency; when encoding substation equipment, the information of the previous frames can be used to predict the content of the current frame, thereby reducing redundant data. By using conditional probability tables and online learning algorithms, the system can quickly adapt to different types of power equipment and environmental conditions. This adaptive capability is particularly important when dealing with diverse scenarios, such as dense substations to open transmission lines, ensuring consistency in coding efficiency across different scenarios. The adaptive probabilistic model update mechanism, especially the application of sliding windows and exponential decay factors, enables the system to adjust the coding strategy in real time to adapt to dynamic changes in video content. This performs well when dealing with changes in lighting conditions such as weather changes or changes in device status due to load changes, ensuring continuous optimization of coding efficiency. The adaptive adjustment of model complexity further enhances the flexibility of the system, by dynamically balancing model accuracy and computational complexity, while optimizing system resource usage while ensuring coding quality. This is particularly beneficial for real-time encoding on edge devices, such as achieving efficient on-site video compression and transmission in drone inspections. The introduction of dynamic verification and model rollback mechanisms provides the system with powerful self-optimization capabilities. This ensures that the coding efficiency can always remain at the best state during long inspection tasks, and can quickly adjust even in the face of unexpected scene changes. The optimization of the entire CABAC process not only improves the compression rate, but also maintains the integrity of key visual information. For example, when encoding the joints of high-voltage lines, the system can allocate more bits to accurately represent possible minor defects or anomalies, while for large areas of uniform background, such as the sky or the ground, a more efficient encoding method can be used. This intelligent bit allocation strategy not only reduces transmission and storage requirements, but also ensures that after decompression, operators or automatic analysis algorithms can obtain high-quality images of key areas.

[0130] By introducing the forward error correction mechanism of Reed-Solomon code and the checksum mechanism at the packet level, the reliability and robustness of the power inspection video compression system during data transmission are improved, making it suitable for dealing with the complex and harsh communication environment commonly seen in power inspection. The application of forward error correction coding, especially the use of Reed-Solomon code, provides a strong error recovery capability for the compressed video data. This error correction capability is particularly important in power inspection scenarios, such as electromagnetic interference near high-voltage lines or unstable signals in remote mountainous areas. It can directly recover the interfered or lost data at the receiving end without the need for retransmission, thereby ensuring the continuity and integrity of the video stream. This is essential for real-time monitoring of the status of power equipment, ensuring that operators can obtain key information in a timely manner, such as flashover of insulators or abnormal swing of lines. The packet segmentation and checksum generation mechanism further enhances the system's anti-interference ability. By segmenting the encoded data block into fixed-size packets and calculating the CRC checksum for each packet, the system can accurately locate transmission errors and retransmit only the erroneous small packets instead of the entire video frame or data block. This fine-grained error detection and recovery strategy not only improves transmission efficiency but also reduces bandwidth requirements, which is particularly important in resource-constrained remote inspection scenarios. For example, when conducting drone inspections of remote transmission lines, most of the video data can be successfully transmitted even when the signal is unstable, and only a small number of damaged data packets need to be retransmitted or recovered. Although the addition of redundant check bits slightly increases the amount of data, it improves the success rate of data recovery. This trade-off is very necessary in ensuring the reliability of critical power facility monitoring data. For example, when monitoring substation equipment, even if individual data packets are lost during transmission, the system can still reconstruct a complete image of the equipment status through redundant information without missing possible signs of failure. This error recovery and robustness enhancement mechanism also improves the system's adaptability under different network conditions. When the network conditions are good, the additional redundant information has little impact on the transmission speed; when the network conditions are poor, these mechanisms can ensure the reliable transmission of critical data. This adaptability is crucial for the stable operation of the power inspection system, because the inspection route may cross a variety of terrains and network environments, from densely populated urban areas to remote mountainous areas. The entire error recovery and robustness enhancement process not only improves the reliability of data transmission, but also indirectly improves the efficiency and effectiveness of the entire power inspection system. For example, it reduces the number of repeated inspections due to data corruption or loss, saving time and resources; at the same time, high-quality, continuous video streaming enables automated analysis algorithms to more accurately identify potential problems and improve the accuracy of fault warnings. This mechanism also enhances the system's performance in emergencies or severe weather conditions.In situations such as typhoons and heavy rains that may cause a sharp decline in communication quality, the system can still ensure the transmission of the most critical equipment status information, providing necessary data support for emergency decision-making and rapid response.

[0131] This embodiment improves the performance and intelligence level of the power inspection video compression system. It not only improves the efficiency of data transmission and storage, but also provides a high-quality data foundation for subsequent video analysis and fault diagnosis, thereby enhancing the ability of remote monitoring and preventive maintenance of the power system, and making important contributions to the safe and efficient operation of the smart grid. Through advanced error correction and data protection technology, the reliability and practicality of the power inspection video compression system in practical applications are improved. It not only ensures the integrity and accuracy of video data in various complex environments, but also provides a more reliable data foundation for the safety monitoring, fault diagnosis and preventive maintenance of the entire power system. This high degree of reliability and robustness enhances the ability of power companies to conduct remote real-time monitoring and rapid response, thereby improving the operational efficiency and safety of the entire power grid, and making important contributions to the development of smart grids and the modern management of power systems.

[0132] like Figure 7 As shown, according to another aspect of the present application, a scene-adaptive video compression method based on natural language guidance specifically includes the following steps:

[0133] Step 1: According to different task application scenarios, set the corresponding task content description natural language text;

[0134] Step 2: Encode the natural language text content input in step 1 using a large language model;

[0135] Step 3: Based on the code generated in step 2 and the input current frame to be compressed x t , predict the foreground and background masks corresponding to different scenes;

[0136] Step 4: Compress the original feature map according to the mask generated in step 3, that is, use element-by-element multiplication to keep the pixel values ​​corresponding to the foreground unchanged and reduce the pixel values ​​of the background to half of the original value.

[0137] Step 5 and steps 1-3 remain unchanged, and the value of the element predicted as the foreground in step 4 is doubled, while the value of the element predicted as the foreground remains unchanged.

[0138] In the encoding stage, step 1 includes:

[0139] Step 11: Give the model guidance by introducing language guidance, so that it can more accurately judge the foreground and background in the feature map. In different scenarios, the model needs to selectively focus on certain parts. Therefore, the language guidance mechanism is introduced to enable the model to learn the characteristics of multiple scenarios at the same time in one training. In this way, the model can integrate the focus points in different scenarios and further improve the accuracy of model prediction.

[0140] Through the language guidance mechanism, the model can be provided with task descriptions for different scenarios, such as mobile communications, video surveillance, video conferencing, etc. Each task description will emphasize the key features and optimization goals of the scenario, guiding the model to learn how to adapt to the compression requirements in different scenarios.

[0141] Step 12: During the training process, the model will receive task descriptions related to specific scenarios and use language guidance to strengthen the learning of specific features. For example, in a video surveillance scenario, the model may be guided to focus on the recognition and tracking of moving objects and the compression of static elements in the background. In a video conferencing scenario, the model may be guided to focus on face recognition and expression compression, as well as blurring the background environment.

[0142] In this way, the model can obtain instructions about the foreground and background from the language guidance of different scenes, and perform feature learning and prediction based on this. In this way, when facing video compression tasks in different scenes, the model can more intelligently adapt to the needs of different scenes and improve compression effects and prediction accuracy.

[0143] Furthermore, the step 2 comprises:

[0144] Step 21: Encode the description of the task in step 1 into tokens using a large language model;

[0145] Step 22. Typically, text-based image generation and manipulation methods use the text encoder of CLIP. However, CLIP is not suitable for degraded user prompts containing little or no visual content. Therefore, Sentence-BERT using plain text is selected. Compared with CLIP, Sentence-BERT is more compact and fast, and can effectively encode the semantics of different user prompts. In the present invention, the BGE-micro-v2 model is used.

[0146] Step 23. In the specific implementation, the text description in step 1 is fed into the text encoder to generate a 768-dimensional token ∈ R 1×768These token vectors contain the semantic information of the task description and serve as a guide for subsequent video compression based on different scenarios. This not only improves the efficiency and accuracy of encoding, but also ensures the model's ability to accurately judge and process the foreground and background in a variety of scenarios. In this way, the task characteristics of different scenarios can be effectively integrated into the model training process, thereby improving the performance of the model in practical applications.

[0147] Furthermore, the step 3 comprises:

[0148] Step 31: Compare the code generated in step 2 with the input current frame x to be compressed. t , predict the foreground and background masks corresponding to different tasks;

[0149] Step 32: For the token encoded in step 2, use the full connection operation to align it with the video compression frame processing part f frame High-dimensional feature extractor E in module high The same dimensions as the output, namely:

[0150] m c =FC(Text token )R 1×768 Align to R 1×feature-dim ;

[0151] Where FC represents the fully connected layer, m c Represents the text encoding vector after alignment through the fully connected layer, Text token Represents a text token, R 1×feature-dim represents the dimension of the vector, where it is aligned from 1×768 to 1× feature dimension;

[0152] For the output text encoding vector m c , and apply it layer by layer to the feature map z t On each layer, that is:

[0153] z t ' = conv(z t m c )+z t ;

[0154] where z t ' represents the feature map after the convolution operation, Conv represents the convolution operation with unchanged channel, and the feature map z t The current frame to be compressed x t After the high-dimensional feature extractor E high generate:

[0155] z t =Ehigh (x t );

[0156] Final feature map z t ' is sent to the mask prediction network Mnet, which is composed of multiple layers of Mblocks. Each Mblock consists of: convolution, normalization and activation function. Among them, two-dimensional convolution, BatchNorm (batch normalization) and LeakyReLU activation function are used. Feature map z t 'After each layer of Mblock, the channel dimension will be halved. After passing the last layer of Mblock, the dimension becomes 1, generating the feature map z t 〞:

[0157] z t =MNet(z t ');

[0158] where z t 〞 represents the final feature map after being processed by the mask prediction network Mnet.

[0159] For values ​​greater than 0.6, it is considered as foreground; otherwise, it is predicted as background:

[0160]

[0161] At this point, a Mask that distinguishes the foreground and background is generated.

[0162] Further, the step 4 comprises:

[0163] Step 41: Based on the mask generated in step 3, apply the mask to the feature map z t In the example, the mask is multiplied with the feature map z using element-by-element multiplication. t Combined to generate the result w t For the area where the value in the Mask is True, that is, the foreground area, the model does not change the element value; for the area where the value in the Mask is False, that is, the background area, the corresponding element value is reduced to half of the original value:

[0164]

[0165] Since the values ​​of the background content are compressed, these background contents will take up less storage space in the subsequent quantization stage, thereby further improving the compression rate of the model without affecting the content of the restored video.

[0166] In the decoding phase, step 5 includes:

[0167] Step 51, the first and third steps of the decoding stage are the same as the encoding stage, and the only difference is in the fourth stage recovery process:

[0168] Step 52: Based on the mask generated in step 3, apply the mask to the feature map z t In the example, the mask is multiplied with the feature map z using element-by-element multiplication. t Combined to generate the result w t For the area where the value in the Mask is True, that is, the foreground area, the model does not change the element value. For the area where the value in the Mask is False, that is, the background area, the corresponding element value is reduced to twice the original value, thereby restoring the pixel value.

[0169]

[0170] like Figure 8 As shown, the main modules of this embodiment include a natural language text encoding module, an intra-frame encoding module, a context encoding module, a motion encoding module, a high-resolution encoder and decoder, a low-resolution encoder and decoder, and an entropy model. Through these modules, the video can be compressed and decompressed in different scenarios, and the scene characteristics and high-dimensional context information are used to improve the efficiency of encoding and decoding. Specifically, for the input uncompressed video sequence, each frame of video is used as the input data of the neural network; the input frame first extracts high-resolution features and low-resolution features through a high-resolution encoder and a low-resolution encoder. The extracted high-resolution features and low-resolution features are combined by element-by-element multiplication to generate a feature map for further processing; the video frame is analyzed by a deep neural network, which includes motion estimation and compensation. The network will automatically learn and predict the motion vector and residual information between each frame and the previous frame. This process is more accurate and adaptable than traditional encoding; for the input natural language text, the text is encoded by a text encoder to obtain the corresponding feature vector. The text feature vector is fused with the deep neural network features corresponding to the video frame through the cross-attention mechanism to obtain a composite feature map that integrates the scene-guided text information; the predicted motion information and residual, as well as the composite feature map, are sent to the quantization module to obtain a quantized feature map. The quantization process is guided by the context coding module and the motion coding module. The quantized feature map is further compressed by the entropy coding module for storage or transmission. It is further quantized and entropy encoded to reduce data redundancy; during decoding, this process is reversed to recover video frames close to the original quality from the received bit stream. The entire video compression framework is optimized through end-to-end training to ensure the best balance between coding efficiency and video quality. It is especially suitable for scenarios such as power inspection, which need to ensure the clarity and real-time performance of the video under limited bandwidth, so as to achieve efficient remote monitoring and analysis.

[0171] This embodiment realizes intelligent video compression in different scenarios by introducing a language-guided mask generation module, improves the efficiency of video compression and the quality of compressed video, and has advantages in specific application scenarios (such as drone inspection of power facilities). The language-guided mask generation module uses a pre-trained language model (such as Sentence-BERT) to encode the text description and generate a vector representing the semantics of the text. Through a neural network, the vector is combined with the video frame features to generate a mask for video compression. The mask is used to perform different processing on the foreground and background areas during the video compression process. The foreground area (such as power facilities in drone inspections) maintains the original resolution to ensure that key details are retained; the resolution of the background area can be reduced to reduce the amount of data.

[0172] Based on the mask-based selective processing method, different compression strategies are used for the foreground and background areas, so that the compressed video can not only ensure the integrity of important information, but also effectively reduce the amount of data and improve transmission and storage efficiency. The model learns the characteristics of multiple scenes at the same time during the training process, and can flexibly adapt to different scene requirements in practical applications.

[0173] The mask generation module combines the mask with the feature map through element-by-element multiplication to generate a new feature map. For areas where the value in the mask is True, that is, the foreground area, the model does not change the element value; for areas where the value in the mask is False, that is, the background area, the corresponding element value is reduced to half of the original. This processing method ensures high-quality compression of important areas while effectively reducing redundant information in background areas.

[0174] By introducing a language-guided mechanism and using the powerful semantic understanding capabilities of the pre-trained language model, the model can accurately identify and process the foreground and background in different scenarios, thereby achieving excellent compression effects in different application scenarios. The language-guided mask generation module and the element-by-element multiplication method jointly achieve intelligent video compression, effectively improve compression efficiency and video quality, and meet video compression requirements in different scenarios.

[0175] This embodiment is based on natural language guidance to achieve high-rate adaptive video compression in different scenarios such as power inspection. With the help of a large language model, according to the characteristics of specific scenarios, the system is provided with preferences and requirements for compression quality and compression ratio by parsing language instructions, and compression parameters and compression ratio are flexibly adjusted to achieve efficient and high-quality video compression effects.

[0176] According to one aspect of the present application, a scene-adaptive video compression system based on natural language guidance includes:

[0177] at least one processor; and,

[0178] a memory communicatively connected to at least one of the processors; wherein,

[0179] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the scene-adaptive video compression method based on natural language guidance as described in any of the above embodiments.

[0180] In the field of power inspection, the present invention can be applied to the compression and transmission process of video data. Through natural language guidance and scene adaptive compression, efficient video transmission and storage can be achieved, and the efficiency of power equipment monitoring and maintenance can be improved. In the field of security, the present invention can be applied to video compression and transmission of video monitoring systems. Through natural language guidance and scene adaptive compression, the compression parameters can be flexibly adjusted according to the specific monitoring scene, efficient video transmission and storage can be achieved, and the performance and efficiency of the monitoring system can be improved. In the field of unmanned driving, the present invention can be applied to the compression and transmission of video data of unmanned vehicles. Through natural language guidance and scene adaptive compression, the compression parameters can be adjusted according to different driving scenes and requirements, efficient video transmission and storage can be achieved, and the performance and safety of unmanned driving systems can be improved. In the field of video conferencing and remote communication, the present invention can be applied to the compression and transmission of video data. Through natural language guidance and scene adaptive compression, the compression parameters can be adjusted according to different communication scenes and bandwidth conditions, efficient video transmission and real-time communication can be achieved, and the experience quality of remote collaboration and communication can be improved.

[0181] The scene-adaptive video compression method based on natural language guidance proposed in the present invention achieves technical effects and performance improvement in specific scenarios such as power inspection. By introducing natural language guidance and scene semantic analysis, the system's ability to understand power inspection scenarios is enhanced. This enables the compression algorithm to dynamically adjust strategies according to different inspection tasks (such as inspecting transmission lines, substation equipment, etc.), achieving unprecedented scene adaptability. For example, when key instructions such as "checking high-voltage line joints" are identified, the system automatically improves the image quality of the relevant area to ensure that subtle fault features are not lost during the compression process. The improved multi-directional adaptive wavelet transform and hexagonal block matching algorithm enhance the system's ability to capture complex power equipment structure and motion features. This is particularly important for accurately compressing equipment with complex geometric shapes such as transmission towers and transformers, ensuring that key structural information can still be retained at low bit rates. The application of feature fusion and foreground and background segmentation technology enables the system to accurately distinguish between key equipment and background areas, achieving differentiated compression. This not only improves the overall compression efficiency, but also ensures high-fidelity compression of power equipment, providing reliable visual data for remote monitoring and fault diagnosis. Adaptive quantization and coding technology further optimizes the bit allocation strategy. The system can dynamically adjust compression parameters according to the importance of the equipment and the potential risk of failure, maximizing the transmission of key information under limited bandwidth. For example, more bits are allocated to the insulator area where there may be faults, while a higher compression ratio is used for the equipment area that is operating normally. Error recovery and robustness enhancement mechanisms improve the reliability of the system in complex power environments. Even in the case of electromagnetic interference or unstable signals, the complete transmission of key monitoring data can be ensured, enhancing the reliability and real-time performance of remote inspections.

[0182] This embodiment not only achieves efficient video compression, but more importantly, it opens up new possibilities for intelligent monitoring and maintenance of power systems. By providing high-quality, low-latency video streams, the accuracy of remote fault diagnosis is improved, emergency response speed is accelerated, and a reliable data foundation is provided for preventive maintenance. The application of this embodiment is expected to significantly improve the safety and efficiency of power grid operations, promote the development of smart grids, and make important contributions to the digital transformation of the power industry.

[0183] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A scene-adaptive video compression method based on natural language guidance, characterized in that: The steps include: S1. Obtain an original video data stream, parse the original video data stream into a continuous video frame sequence; obtain a natural language guidance text related to the scene, construct a text feature vector, and calculate a scene relevance score based on the text feature vector and a predefined scene type template; Based on the scene relevance score, determine the type of the current scene; combine the type of the current scene with the natural language guidance text to generate an enhanced scene description text; The original video data stream includes the appearance of the high-voltage power line, the surrounding environment and the possible fault point characteristics; S2, based on the enhanced scene description text, use the rule-based scene classification system to perform in-depth analysis and generate a scene feature vector; Combine the scene feature vector with the pre-trained large language model to generate an optimization strategy for the current scene; Based on the video frame sequence and the optimization strategy, a multi-scale analysis method based on wavelet transform is used to extract local and global features; Reconstruct high-resolution features and low-resolution features based on local and global features; combine high-resolution features and low-resolution features to generate a comprehensive feature map; S3, based on the comprehensive feature map and the video frame sequence, use the hexagonal block matching algorithm to perform motion estimation, generate motion vectors and residual information; encode and compress the motion vectors and residual information, and output compressed motion information data; based on the comprehensive feature map and the compressed motion information data, perform feature fusion to generate a composite feature map; S4, based on the composite feature map and optimization strategy, the improved watershed algorithm is used to generate foreground and background masks; Based on the foreground and background masks, the composite feature map is adaptively compressed and the compressed feature data is output; S5, based on the compressed feature data, further data compression is performed using context-adaptive binary arithmetic coding to output a final compressed data bit stream; Based on the final compressed data bit stream, an error recovery mechanism and robustness enhancement measures are adopted to output a compressed data packet with error resilience.

2. The scene-adaptive video compression method based on natural language guidance according to claim 1, characterized in that: Step S1 is further as follows: S11, obtaining an original video data stream through a power inspection device, preprocessing the original video data stream, removing damaged or incomplete data packets, and obtaining a preprocessed video data stream; Extract metadata information based on the preprocessed video data stream, including resolution, frame rate and encoding format; Based on the metadata information, the original video data stream is parsed into a continuous sequence of video frames; S12, obtaining natural language guidance text related to the current scene from a user input or a preset scene description library, preprocessing the natural language guidance text, including removing special characters, unifying upper and lower cases and segmenting words, to obtain a preprocessed natural language guidance text; using a pre-trained natural language processing model to perform part-of-speech tagging and named entity recognition on the preprocessed natural language guidance text to obtain a tagged text; extracting keywords and phrases from the tagged text to construct a text feature vector; performing similarity calculation based on the text feature vector and a predefined scene type template to obtain a similarity calculation result; based on the similarity calculation result, preliminarily determining the type of the current scene to obtain a preliminarily determined scene type; S13, based on the preprocessed natural language guide text, perform deep semantic analysis to extract core semantic features; match the core semantic features with entries in a pre-built scene feature dictionary and calculate a matching score; Based on the matching score and a preset weight system, the relevance score of each scene type in the initially determined scene types is calculated, and the scene type with the highest relevance score is selected as the type of the current scene finally determined; S14, based on the final determined type of the current scene, selecting a corresponding template from a preset scene description template library to obtain a selected scene description template; The preprocessed natural language guidance text is fused with the selected scene description template to generate enhanced scene description text.

3. The scene-adaptive video compression method based on natural language guidance according to claim 2, characterized in that: Step S2 is further as follows: S21. Based on the enhanced scene description text, use an improved word frequency-inverse document frequency algorithm to calculate the weight of each word; based on the calculated weight, select a corresponding scene feature vector from a preset weight system; S22. Use the pre-trained Sentence-BERT model to encode the enhanced scene description text to generate a semantic vector; concatenate the semantic vector with the scene feature vector to form an enhanced semantic representation; use a multi-layer perceptron to reduce the dimension of the enhanced semantic representation to generate a compressed semantic vector; apply the attention mechanism to calculate the importance weights of each dimension in the compressed semantic vector; Based on the importance weights, the compressed semantic vectors are weighted summed to obtain the final scene semantic features; S23, inputting the scene semantic features into the pre-trained large language model to generate a preliminary compression strategy description for the current scene; matching the preliminary compression strategy description with a predefined compression rule library to obtain a matching result; Based on the matching results, a compression parameter combination is selected from the compression rule library; based on the compression parameter combination and the preliminary compression strategy description, an optimization strategy including initialization parameters is constructed; S24, applying an improved discrete wavelet transform to decompose each frame image of the video frame sequence to obtain multi-scale sub-band coefficients, including low-frequency sub-band coefficients and high-frequency sub-band coefficients; applying a high-resolution encoder to the low-frequency sub-band coefficients to generate a global feature vector; Apply a low-resolution encoder to the high-frequency subband coefficients to generate a local texture feature vector; use an adaptive threshold method to perform soft threshold processing on the multi-scale subband coefficients to obtain denoised coefficients; uniformly quantize the denoised coefficients to obtain discretized coefficient values; based on the discretized coefficient values, use run-length coding to encode and obtain a compressed coefficient sequence; S25. Reconstruct the high-resolution image and the low-resolution image using an improved inverse discrete wavelet transform based on the global feature vector, the local texture feature vector and the compressed coefficient sequence; apply a convolutional neural network to the high-resolution image to obtain high-resolution features; Apply a lightweight convolutional neural network to low-resolution images to obtain low-resolution features; The attention mechanism is used to calculate the weight coefficients of high-resolution features and low-resolution features. Based on the weight coefficients, the high-resolution features and low-resolution features are weighted and summed to obtain a comprehensive feature map.

4. The scene-adaptive video compression method based on natural language guidance according to claim 3, characterized in that: Step S3 is further as follows: S31, dividing the current frame and the reference frame of the video frame sequence into overlapping hexagonal blocks; based on the comprehensive feature map, for each hexagonal block in the current frame, performing block matching within the search window of the reference frame to obtain a predetermined number of candidate matching positions; calculating the matching cost of each candidate matching position, and selecting the position with the smallest matching cost as the best matching position; calculating the displacement between the current block and the best matching position to obtain an initial motion vector; using a half-pixel precision interpolation method to refine the initial motion vector to obtain an optimized motion vector; S32, performing motion compensation on the reference frame based on the optimized motion vector to generate a predicted frame; Calculate the difference between the predicted frame and the current frame to get the residual data; Applying discrete cosine transform to the residual data to obtain the frequency domain residual coefficient; quantizing the frequency domain residual coefficient using an improved adaptive quantization method to obtain the quantized residual coefficient; S33, performing differential encoding based on the optimized motion vector to obtain a differential motion vector; Performing entropy coding on the differential motion vector using Huffman coding to obtain compressed motion vector data; applying run-length coding to the quantized residual coefficient to obtain compressed residual data; Further compressing the compressed residual data using context-adaptive binary arithmetic coding to obtain final compressed residual data; merging the compressed motion vector data and the final compressed residual data to generate final motion information data; S34, upsampling the scene semantic features to make their dimensions match the comprehensive feature map; decoding the motion information data into a motion feature map, and using a linear interpolation method to adjust the size of the motion feature map to make it consistent with the comprehensive feature map; outputting the aligned scene semantic features, the comprehensive feature map, and the motion feature map; S35, based on the aligned scene semantic features, comprehensive feature map and motion feature map, the feature weight coefficient is calculated using the attention mechanism; Based on the feature weight coefficient, weighted summation is performed to obtain preliminary fusion features; the preliminary fusion features are processed using a 3D convolutional neural network to obtain processed features; the aligned scene semantic features, comprehensive feature maps, and motion feature maps are added to the processed features using residual connections to obtain the final composite feature map.

5. The scene-adaptive video compression method based on natural language guidance according to claim 4, characterized in that: Step S4 is further as follows: S41, applying Gaussian filtering to the composite feature map to obtain a filtered image; based on the filtered image, calculating the gradient amplitude to obtain a gradient map; using morphological operations to process the gradient map to obtain a processed gradient map; S42, based on the parameters in the optimization strategy, determining the threshold and seed point selection strategy of the watershed algorithm; applying the distance transform on the processed gradient map to obtain a distance map; using the H-minima transform to suppress the local minimum in the distance map to obtain a suppressed distance map; based on the suppressed distance map, selecting the seed point according to the threshold and the seed point selection strategy; Starting from the seed point, the improved watershed algorithm is executed to obtain the preliminary segmentation result; S43, based on the preliminary segmentation result, using a region merging algorithm to merge similar small regions to obtain a merged segmentation result; Based on the merged segmentation results, an edge-preserving filter is used to smooth the region boundaries to obtain a smooth segmentation result; Based on the preset scene semantic information, adjust the division of foreground and background in the smooth segmentation result to generate a binary mask; S44, dividing the composite feature map into a foreground area and a background area based on the binary mask; Apply fine-grained quantization to the foreground area to obtain foreground area data; Apply coarse-grained quantization to the background area to obtain background area data; The entropy coding method is used for the foreground area data, and the run-length coding method is used for the background area data to generate a coded bit stream, that is, compressed feature data.

6. The scene-adaptive video compression method based on natural language guidance according to claim 5, characterized in that: Step S5 is further as follows: S51, constructing a context probability model based on the compressed feature data; encoding the compressed feature data using a context adaptive binary arithmetic coding algorithm based on the context probability model to generate a compressed data bit stream; S52, encoding the compressed data bit stream using a Reed-Solomon code to generate a redundant check bit; combining the compressed data bit stream and the redundant check bit to form a data block with error correction capability; S53, dividing the data block with error correction capability into data packets of a predetermined size, and calculating a checksum of a cyclic redundancy check for each data packet; A checksum is added to the header of each packet to assemble a compressed packet with error resilience.

7. The scene-adaptive video compression method based on natural language guidance according to claim 6, characterized in that: It also includes optimizing the comprehensive feature map of step S25, specifically: S25a, applying a non-local means algorithm to process the comprehensive feature map to obtain a denoised feature map; S25b. Based on the denoised feature map, adaptive histogram equalization is used to adjust the contrast to obtain an optimized comprehensive feature map.

8. The scene-adaptive video compression method based on natural language guidance according to claim 6, characterized in that: Step S2 also includes: S261, based on the parameters in the optimization strategy, adaptively quantize the comprehensive feature map to obtain a quantized feature map; use a content-based adaptive filter to process the quantized feature map to obtain a smoothed feature map; S262, based on the smoothed feature map, using a local quality assessment method based on structural similarity to generate a quality map of the feature map; based on the quality map, identifying key areas and generating a key area mask; S263, based on the key area mask, using a local contrast enhancement algorithm to obtain enhanced key area features; based on the enhanced key area features, using an image segmentation-based region growing algorithm to obtain optimized edge and contour information; S264. Based on the optimized edge and contour information, a sparse representation method is used to obtain a sparse coefficient matrix; the sparse coefficient matrix is ​​reduced in dimension using the principal component analysis method to obtain a final low-dimensional feature representation.

9. The scene-adaptive video compression method based on natural language guidance according to claim 6, characterized in that: The process also includes optimizing the motion information data in step S33, specifically: S33a, based on the parameters in the optimization strategy, using a rate-distortion optimization algorithm, performing adaptive bit allocation on the motion information data to obtain a bit allocation result; S33b, based on the bit allocation result, adopting the sliding window technology, re-encoding the motion information data to obtain optimized motion information data; S33c, based on the optimized motion information data, perform bit stream packaging to obtain a final bit stream format, and generate corresponding decoding instructions.

10. A scene-adaptive video compression system based on natural language guidance, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the scene-adaptive video compression method based on natural language guidance as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Panoramic background prediction video coding and decoding method

    CN102333221A

  • Hierarchical motion estimating method and device in allusion to ultrahigh-definition video application

    CN107872674A

  • Video encoding method, video decoding method and related device

    CN108632625A

  • Texture image defect classification system and method based on multi-resolution feature fusion

    CN113989542A

  • Image generation method and device, equipment and medium

    CN116977774A

Cited By

  • Balanced control method and system for partition backlight source and storage medium

    CN119152816A

  • Monitoring video compression processing method and device

    CN119211560A

  • Three-dimensional assembly method based on single-view image and large language model guidance

    CN119579896A

  • Visual laser data feature compression and extraction method and system

    CN119919747A

  • Monocular depth estimation method and system based on text guidance and multi-scale fusion

    CN119941816A