A semantic contour generation type image coding method based on key point optimization

Abstract

The present application relates to the field of image and video compression technology, and more particularly to a semantic contour generation type image coding method based on key point optimization, comprising obtaining an image to be processed and inputting a visual language model and an instance segmentation model respectively to generate an overall text description and a complete contour mask; inputting the complete contour mask into a VQ-VAE encoder to output a codebook index sequence set; for the complete contour mask, a differentiable model based on Gumbel-Softmax is used to adaptively select a group of key points; a VQ-VAE decoder generates a preliminary reconstructed contour according to the codebook index sequence set; a lightweight optimization network performs geometric correction and refinement on the preliminary reconstructed contour based on the key points to obtain an optimized contour; and the optimized contour and the overall text description are taken as control conditions to generate a high-quality reconstructed image through a conditional generation model; and the present application replaces a non-structured global pixel edge map with a structured contour set to eliminate redundant edge information in the background and the object interior.

Description

Technical Field

[0001] This invention relates to the field of image and video compression technology, and in particular to a semantic contour generative image coding method based on key point optimization. Background Technology

[0002] With the explosive growth of image / video data, efficient compression techniques for machine analysis and semantic understanding have become a research hotspot, as traditional compression frameworks aimed at pixel fidelity are no longer sufficient. Against this backdrop, semantic compression (or human-understandable compression) has emerged as a new direction, with its core objective being to preserve and transmit high-level semantic information of images at extremely low bitrates.

[0003] The paper "Cross Modal Compression: Towards Human-comprehensible Semantic Compression" systematically proposes a cross-modal compression (CMC) framework. This framework advocates transforming and compressing highly redundant visual data (such as images) into a compact and human-comprehensible domain (such as text, sketches, or semantic graphs). Specifically, it implements an "Image-Text-Image (ITI)" paradigm: the encoder generates a text description using an image description model, while the decoder reconstructs the image from the text using a text-to-graph model. This work verifies that ultra-high compression ratios can be achieved by transmitting high-level semantic information such as text, and surpasses traditional encoders like JPEG in semantic fidelity. However, this scheme has significant limitations: it compresses the entire image into a single text description. While the text is highly abstract, it severely lacks the ability to describe the spatial structure and geometric details of the image, resulting in the loss of crucial structural information such as the precise shape and relative position of objects in the reconstructed image, leading to low quality of detail reconstruction.

[0004] To compensate for the lack of structural information in plain text compression, the paper "Text + Sketch: ImageCompression at Ultra Low Rates" proposes a hybrid compression scheme of "text + sketch". This scheme transmits an optimized text prompt along with an additional compressed image sketch (e.g., a full-image edge map extracted using a HED edge detector). At the decoding end, a conditional generation model such as ControlNet is used to reconstruct the image by combining the text and sketch. This method significantly improves the structural preservation and visual quality of the reconstructed image at extremely low bitrates. However, this scheme still has several key drawbacks: First, the sketch used is derived from a general edge detector, generating a pixel-level, unstructured global edge map that fails to distinguish between foreground semantic objects and irrelevant background, thus containing a large number of noisy edges that are redundant or even harmful to semantic understanding. Second, the compression of this sketch uses a general encoder designed for natural images, which fails to optimize for the structured and sparsity characteristics of the contour signal, leaving room for improvement in compression efficiency. Finally, this scheme lacks a mechanism to perform precise geometric optimization and correction of the recovered coarse contour at the decoding end using high-level semantic information.

[0005] In summary, while existing technical solutions (i.e., "text + sketch" compression) improve structural reconstruction by introducing edge information, the extraction, representation and reconstruction processes of contour information are not deeply integrated with the semantic content of the image, resulting in redundancy of contour information and limited reconstruction accuracy, thus restricting the potential to achieve higher semantic and structural fidelity at extremely low bit rates. Summary of the Invention

[0006] To address the above problems, this invention provides a semantic contour generative image coding method based on key point optimization, comprising the following steps:

[0007] S1. Obtain the image to be processed and input it into the visual language model and instance segmentation model respectively to generate an overall text description and a complete contour mask;

[0008] S2. Input the complete contour mask into the VQ-VAE encoder and output a set of codebook index sequences;

[0009] S3. For each complete contour mask, a set of key points is adaptively selected using a differentiable model based on Gumbel-Softmax;

[0010] The S4.VQ-VAE decoder generates a preliminary reconstruction profile based on the codebook index sequence set;

[0011] S5. The lightweight optimization network performs geometric correction and refinement on the initial reconstructed contour based on key points to obtain the optimized contour;

[0012] S6. Using optimized contours and overall text descriptions as control conditions, a conditional generation model is used to generate high-quality reconstructed images.

[0013] The beneficial effects of this invention are:

[0014] This invention replaces the unstructured global pixel edge map with a structured contour set, fundamentally eliminating redundant edge information in the background and within objects. Combined with efficient codebook indexing, the code rate of contour branches is reduced by orders of magnitude.

[0015] Despite the significant reduction in bit rate, the transmitted contours themselves have clear semantic object boundaries, providing higher quality and less noise spatial control information for the decoding end's generation model. This ensures the recognizability of each object and the integrity of its basic structure in the reconstructed image even at extremely low total bit rate. Attached Figure Description

[0016] Figure 1 This is a flowchart of a semantic contour generation image coding method based on key point optimization according to the present invention.

[0017] Figure 2 This is the quantitative result of the embodiments of the present invention on the COCO dataset;

[0018] Figure 3 This is the quantitative result of the embodiment of the present invention on the VOC dataset. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 To address the issues of high contour information redundancy and lack of semantic guidance in existing "text + edge map" compression methods, this invention proposes a semantic contour generative image coding method based on keypoint optimization. This method aims to guide contour feature extraction by introducing image semantic category information and combining it with a differentiable keypoint selection and optimization mechanism to construct an image compression scheme with extremely low bitrate while accurately preserving the semantic and structural information of multiple foreground objects in the image, thereby significantly improving the quality of reconstructed images at extremely low bitrates.

[0021] This invention discloses a semantic contour generation image coding method based on key point optimization, which mainly consists of two parts: encoding and decoding. The method includes the following steps:

[0022] At the encoding end:

[0023] S1. Acquire the image to be processed and input it into the visual language model and the instance segmentation model respectively to generate an overall text description and a structured contour set; the structured contour set includes multiple complete contour masks.

[0024] Preferably, embodiments of the present invention employ a pre-trained visual language model, such as CLP, to generate a complete text description of the image to be processed.

[0025] Preferably, in order to eliminate the edge information redundancy problem that exists when using pixel-level, unstructured global edge maps in the prior art, this embodiment of the invention proposes a full object contour parsing method, that is, by using semantic segmentation or object detection technology, the complete contours of all foreground semantic objects in the image to be processed are identified and extracted to form a structured, object-specific contour set (i.e., a complete contour mask), thereby replacing the non-semantic global pixel edge map.

[0026] In one specific embodiment, Mask R-CNN is used as an instance segmentation model to extract the complete contour mask of each foreground semantic object in the image to be processed, forming a structured contour set.

[0027] S2. Input the structured contour set into the VQ-VAE encoder and output the codebook index sequence set.

[0028] Preferably, the embodiments of the present invention employ a trained vector quantization autoencoder (VQ-VAE) for processing, including:

[0029] S21. For the complete contour of each foreground semantic object in the complete contour mask, the VQ-VAE encoder is used to generate the corresponding contour features;

[0030] S22. Each contour feature is vector quantized and mapped into a corresponding extremely compact codebook index sequence, forming a codebook index sequence set for transmission, thus achieving efficient semantic compression of contour information;

[0031] Among them, the VQ-VAE encoder introduces semantic category constraints during training. Its encoding process is guided by the semantic category information of the image, thereby learning to extract contour features that are highly correlated with the semantic category information of the image.

[0032] S3. For the complete contour mask, a set of sparse key points is adaptively selected using a differentiable model based on Gumbel-Softmax to represent the main geometric structure of the contour.

[0033] Preferably, the adaptive selection of a set of sparse key points using a differentiable model based on Gumbel-Softmax includes:

[0034] S31. From the complete outline mask Extract the pixels located on the contour to form a candidate point set. ;in, H represents the total number of candidate points, and H and W represent the image height and width. This represents the i-th candidate point. This indicates that the i-th candidate point is within the complete contour mask. The coordinates in the diagram.

[0035] S32. Obtain candidate points using a lightweight scoring network. initial score , can be represented as:

[0036]

[0037] In the formula, f score ( ) represents a lightweight scoring network, and S represents the score image. .

[0038] The lightweight scoring network assigns a score representing the importance of each candidate point, tending to allocate higher scores to locations that are more sensitive to shape, such as contour inflection points, areas with significant curvature changes, and boundary areas prone to shifting. These scores are not manually set but are gradually learned through subsequent training objectives.

[0039] If after this step, directly apply the initial score... Do Performing Top-N point selection operations faces the problem of "non-differentiability": once point selection becomes a hard decision, gradients cannot be backpropagated to the lightweight scoring network during training, making it difficult for the "adaptive" selection mechanism to be effectively learned. To address this, this invention introduces Gumbel-Softmax.

[0040] S33. For each candidate point initial score Normalization yields the selection probability. Gumbel noise and temperature parameter τ are introduced to obtain the soft selection weights for each candidate point. , is represented as:

[0041]

[0042]

[0043]

[0044] In the formula, M represents the number of candidate points. The Gumbel noise represents the i-th candidate point. Indicates a uniform distribution Random numbers sampled from the middle.

[0045] Specifically, here This can be understood as the probability that the i-th candidate point is selected as a key point. It is a continuously differentiable value, and therefore can be used for gradient backpropagation during training. When the temperature parameter τ is large, The distribution is relatively smooth, and the model will explore a wider range of points; as τ gradually decreases... The distribution becomes increasingly sharp, eventually approaching a one-hot form, making the selection process gradually more deterministic. This invention employs a temperature annealing strategy during training, enabling the selection of key points to gradually transition from "exploring effective locations" to "stable and determined point selection."

[0046] S34. Select N candidate points to form a set of key points based on the soft selection weights.

[0047] Preferably, to obtain N key points, this invention employs repeated sampling (or equivalent multiple selections) combined with a neighborhood suppression mechanism during the training phase to prevent all key points from clustering in local areas, thereby ensuring that the key points can cover the entire outer contour. During the inference phase, a hard selection (e.g., taking argmax) is directly used to obtain the final set of key points.

[0048] Once the key points are determined, the encoding end only needs to transmit the coordinates of these key points. Since the number of key points N is very small, the additional overhead is manageable. If each key point coordinate is... and In bit representation, the additional bit overhead of the key point can be expressed as:

[0049]

[0050] While this overhead isn't zero, it's a very small percentage compared to the number of bits in the structure graph and image content itself. Its benefit lies in making the structural conditions more stable, which is crucial for subsequent generation and reconstruction tasks.

[0051] On the decoding end:

[0052] The S4.VQ-VAE decoder generates a preliminary reconstruction profile based on the codebook index sequence set.

[0053] S5. Based on key points, a lightweight contour correction network performs geometric correction and refinement on the initially reconstructed contour to obtain an optimized contour.

[0054] Preferably, the lightweight contour correction network performs geometric correction and refinement of the initially reconstructed contour based on key points as follows:

[0055] For each key point, use it as an alignment anchor point in the initial contour reconstruction. Above the radius Perform a local search and use the edge point closest to the current key point as the matching point. , is represented as:

[0056]

[0057] In the formula, Indicates key points Centered on, with radius The local search neighborhood, |·|² represents the Euclidean distance, used to measure candidate edge points. With key points Spatial distance between them Indicates preliminary contour reconstruction In position The pixel value at that location, when When this occurs, it indicates that the location is an edge point. Represents local neighborhood Candidate edge points that satisfy the edge constraint conditions.

[0058] For each key point Calculate its matching point offset between , is represented as:

[0059]

[0060]

[0061] In the formula, Indicates matching point coordinates These represent the horizontal and vertical coordinate offsets, respectively.

[0062] offset It reflects the offset of the initial reconstructed contour obtained from decoding at key locations.

[0063] offset Diffusion into the neighborhood forms a smooth displacement field Then, the initial reconstructed outline was... Perform warp processing to obtain optimized contours , is represented as:

[0064]

[0065] In this way, the structural profile is pulled back to a more reasonable position at critical locations, and because the displacement field is smooth, the structure will not experience abrupt fractures. The final result is... The outer contour is closer to the original contour at the corners and key boundary positions, thus providing more reliable structural conditions for subsequent model generation.

[0066] Preferably, to ensure that "adaptive keypoint selection" is not merely a module based on empirical interpretation, but rather a truly trainable and verifiable mechanism, this invention places keypoint selection and structural correction within the same training objective. Since contour pixels constitute a small proportion of the entire image and exhibit significant class imbalance, pixel-level loss alone is easily dominated by the background. Therefore, this paper employs a structural loss method combining BCE and Dice. :

[0067]

[0068]

[0069]

[0070]

[0071] In the formula, Indicates the smoothing term. Indicates the weighting coefficient. Indicates BCE loss, This represents the Dice loss, where (x, y) represents the pixel coordinates. Represents the complete outline mask. Indicates optimized contour, Represents the original complete outline mask With optimized contour The degree of overlap; express and The corresponding position is simultaneously the sum of the overlap of edge pixels; This represents the sum of edge pixels in the original complete contour mask E; This represents the sum of edge pixels in the optimized contour.

[0072] In this structural loss, the BCE loss provides a stable pixel-level learning signal. The loss function focuses more on the overall overlap metric of the boundary region, which effectively alleviates the class imbalance problem caused by sparse contours. Keypoint selection participates in the structural correction process through Gumbel-Softmax soft weights, thereby improving the structural loss. The gradients can be backpropagated to the lightweight scoring network. This mechanism drives the model to adaptively learn during training which keypoints to select to minimize the corrected structural error. In other words, the selection of keypoints is adaptively adjusted around the goal of minimizing structural error, ultimately forming the set of keypoints that are most effective in correcting the external contour.

[0073] S6. Using optimized contours and overall text descriptions as control conditions, a high-quality reconstructed image is generated through a conditional generative model (ControlNet).

[0074] Example 1

[0075] For an indoor scene image containing "a cat sitting on a sofa", the method proposed in this invention is used for compression, including:

[0076] The CLIP model was used to generate the following overall text description:

[0077] "loire cessna skimflooding islands singapore malaysian save threatsrockies, fans photo georges laghpig".

[0078] Meanwhile, Mask R-CNN is used to perform instance segmentation on the image, extracting the complete contour masks of two foreground semantic objects, "cat" and "sofa".

[0079] The complete outline masks of "cat" and "sofa" are input into the VQ-VAE encoder trained with semantic category conditional constraints, and the codebook index sequence corresponding to each foreground semantic object is output and transmitted to the decoder.

[0080] At the encoding end, a differentiable model based on Gumbel-Softmax is also required to adaptively select a set of key points (such as contour turning points of ears, limbs, tail, etc.) from the complete contour mask. The key point information is transmitted together with the codebook index sequence.

[0081] The decoder reconstructs a preliminary outline based on the received codebook index sequence. Then, using keypoint information, a lightweight optimization network performs geometric corrections on the "cat" outline (such as correcting ear shape and tail curve). The optimized outline and text description are input into ControlNet to generate the final reconstructed image.

[0082] Example 2

[0083] For a street scene image containing "pedestrians, cars, and traffic lights", the method proposed in this invention is used for compression, including:

[0084] The CLIP model was used to generate the following overall text description:

[0085] "amatvit deciding imports patio relaxing syrah constructing emmysmount anne avec reflections instructions redo president".

[0086] Meanwhile, Mask R-CNN is used to segment the image into instances, extracting the complete contours of "pedestrians", "cars" and "traffic lights" to form a complete contour mask.

[0087] Complete contours of different categories are input into the same VQ-VAE encoder. The encoding process is controlled by category label conditions, learns the contour features of the category, and outputs the codebook index sequence corresponding to each foreground semantic object.

[0088] Extract key points (such as wheels, windows, and roof corners) from the outline of a "car", extract key points (such as head and limb joints) from the outline of a "pedestrian", and extract key points (such as outer contour corners) from the outline of a "traffic light" to form a set of key points.

[0089] The decoding end optimizes the contours of each category, especially the contour structure of "car". Combined with text descriptions, it generates street scene reconstruction images with accurate structure and clear semantics.

[0090] Example 3

[0091] This invention uses the COCO2014 and VOC datasets to test the overall system performance. The method of this invention is executed on each image, encompassing the entire process of semantic extraction, contour parsing, contour compression, keypoint extraction, contour optimization, and image generation.

[0092] Calculate the total bitrate of the transmission, which is the text description bitrate + contour index bitrate + key point bitrate.

[0093] The semantic fidelity of the reconstructed images was evaluated using human evaluation and semantic similarity metrics.

[0094] Use structural similarity metrics to assess contour preservation capability.

[0095] Figures 2-3 The results are quantitative on the COCO and VOC datasets. As shown in the figure:

[0096] Compared to similar advanced semantic compression schemes (PICS), this invention achieves significant bitrate savings;

[0097] Compared to basic semantic compression schemes (PIC), this invention achieves a significant improvement in visual quality at the cost of a small bit rate.

[0098] Compared to the traditional pixel compression paradigm (JPEG), this invention achieves comparable semantic-aware quality with a bitrate that is 1-2 orders of magnitude lower.

[0099] In summary, the "semantic-guided full object contour extraction and key point optimization" technical approach proposed in this invention achieves the dual goals of ultra-high compression ratio and high semantic fidelity.

[0100] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A keypoint-based optimization-based semantic contour generation-based image encoding method, characterized by, Includes the following steps: S1. Obtain the image to be processed and input it into the visual language model and instance segmentation model respectively to generate the overall text description and complete contour mask; S2. Input the complete contour mask into the VQ-VAE encoder and output a set of codebook index sequences; S3. For the complete contour mask, a set of key points is adaptively selected using a differentiable model based on Gumbel-Softmax; The S4.VQ-VAE decoder generates a preliminary reconstruction profile based on the codebook index sequence set; S5. The lightweight contour correction network performs geometric correction and refinement on the initial reconstructed contour based on key points to obtain an optimized contour. S6. Using optimized contours and overall text descriptions as control conditions, a high-quality reconstructed image is generated through a conditional generation model.

2. The keypoint-based semantic contour generation image coding method according to claim 1, characterized in that, Mask R-CNN is used as the instance segmentation model to extract the complete contour of each foreground semantic object in the image to be processed, forming a complete contour mask.

3. The semantic contour generative image coding method based on key point optimization according to claim 1, characterized in that, Step S2 includes: S21. For the complete contour of each foreground semantic object in the complete contour mask, the VQ-VAE encoder is used to generate the corresponding contour features; S22. Each contour feature is vector quantized and mapped to a corresponding codebook index sequence, forming a set of codebook index sequences; Among them, the VQ-VAE encoder introduces semantic category constraints during training. Its encoding process is guided by the semantic category information of the image, thereby learning to extract contour features that are highly correlated with the semantic category information of the image.

4. The semantic contour generative image coding method based on key point optimization according to claim 1, characterized in that, The adaptive selection of a set of sparse keypoints using a differentiable model based on Gumbel-Softmax includes: S31. Extract the pixels located on the contour from the complete contour mask to form a candidate point set. ; S32. Obtain candidate points using a lightweight scoring network. initial score ; S33. For each candidate point initial score Normalization yields the selection probability. Gumbel noise and temperature parameter τ are introduced to obtain the soft selection weights for each candidate point. ; S34. Select N candidate points to form a set of key points based on the soft selection weights.

5. The semantic contour generative image coding method based on key point optimization according to claim 4, characterized in that, Step S33 specifically includes: ， ， ， In the formula, M represents the number of candidate points. The Gumbel noise represents the i-th candidate point. Indicates a uniform distribution Random numbers sampled from the middle.

6. The semantic contour generative image coding method based on key point optimization according to claim 1, characterized in that, The lightweight contour correction network performs geometric correction and refinement of the initially reconstructed contour based on key points as follows: For each key point, use it as an alignment anchor point in the initial contour reconstruction. Above the radius Perform a local search and use the edge point closest to the current key point as the matching point. ; For each key point Calculate its matching point The offset between; The offset is diffused into the neighborhood to form a smooth displacement field, and then the initial reconstructed contour is warped to obtain the optimized contour.

7. The semantic contour generative image coding method based on key point optimization according to claim 1, characterized in that, Introducing structural loss during training Optimize the adaptive selection process of key points, structural loss Represented as: ， ， ， ， In the formula, Indicates the smoothing term. Indicates the weighting coefficient. Indicates BCE loss, This represents the Dice loss, where (x, y) represents the pixel coordinates. Represents the complete outline mask. Indicates optimized contour, express and The degree of overlap; express and The corresponding position is simultaneously the sum of the overlap of edge pixels; This represents the sum of edge pixels in the complete contour mask; This represents the sum of edge pixels in the optimized contour.

Images