Semantic driven image reconstruction method and apparatus based on edge and color assistance

Abstract

The application relates to a semantic-driven image reconstruction method and device based on edge and color assistance, and relates to the technical field of image processing. The method comprises the following steps: a transmitting end performs three-way parallel feature extraction on an original image in terms of semantics, edges and colors, and encodes the features respectively to output semantic encoding sequences, edge encoding sequences and color encoding sequences; according to header information, each encoding sequence is fused to obtain a fused encoding sequence; the fused encoding sequence is subjected to channel encoding to output a channel encoding sequence; a receiving end acquires the channel encoding sequence and decodes the channel encoding sequence to output a decoded sequence; according to the header information, the decoded sequence is subjected to semantic recovery, edge recovery and color recovery and decoding to obtain semantic decoding information, edge decoding information and color decoding information; and a conditional diffusion model is constructed, and each decoding information is input into the conditional diffusion model for processing to output a reconstructed image. The application can improve semantic accuracy, structural fidelity and color consistency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a semantically driven image reconstruction method and apparatus based on edge and color assistance. Background Technology

[0002] With the explosive growth of communication data volume, resource-constrained scenarios such as satellite communication, low-altitude communication, and deep-space communication are increasingly common, making the improvement of information transmission efficiency a critical bottleneck that communication systems urgently need to overcome. Traditional communication systems are based on Shannon's information theory, with the core objective of achieving bit-level lossless or tolerable distortion transmission under channel capacity constraints. While this ensures transmission fidelity and reliability, it only focuses on the information transmission at the signal level, completely ignoring the "semantic" layer of information content, that is, the actual meaning carried by the original information. In modern communication environments, the proportion of high-dimensional multimedia data such as images and videos continues to increase. Traditional compression and transmission mechanisms based on pixels or transform domain coefficients are gradually approaching the theoretical limit of rate distortion. When communication systems are under conditions of low signal-to-noise ratio or limited bandwidth, these traditional solutions cannot simultaneously achieve compression efficiency and semantic fidelity, often resulting in significant semantic distortion or loss of key information after image restoration.

[0003] To overcome the limitations of traditional communication theory, semantic communication, as a novel communication paradigm oriented towards "meaning transmission," has received widespread attention in recent years. Its core idea originates from Weaver's three-layer problem of communication proposed in 1949, with the "semantic layer" focusing on "how the transmitted symbols accurately convey the expected meaning?" Driven by deep learning and artificial intelligence technologies, semantic communication has gradually moved from theoretical concepts to engineering implementation. Typical methods include using semantic segmentation networks to extract key object and scene semantics from images, transmitting only high-level semantic labels or embedding vectors, and then reconstructing the original image at the receiving end using generative models such as generative adversarial networks (GANs) and diffusion models. Existing research has shown that this semantically driven communication method can effectively reduce data volume and maintain high semantic consistency in tasks such as remote sensing, low-light conditions, and real-time video transmission.

[0004] However, existing semantic-driven image reconstruction schemes have two main limitations. First, their ability to restore semantic information is significantly limited. Semantic segmentation results can only describe "what it is," but lack crucial spatial structure and visual detail information such as "where it is," "how it is shaped," and "how it is colored." Images reconstructed solely based on semantic labels may contain the correct object category, but they exhibit significant deviations in structural layout, edge reproduction, and color restoration, making it difficult to meet the high requirements of geometric and photometric fidelity in scenarios such as medical image diagnosis, remote sensing geological analysis, and industrial quality inspection. Second, there is currently a lack of a unified semantic accuracy evaluation system. Image reconstruction quality is typically evaluated using metrics such as SSIM and FID, which focus on pixel or feature distribution similarity. The system fails to effectively reflect the semantic accuracy of generated images. Although some studies have introduced the IoU metric from semantic segmentation as a supplement, a quantitative standard for "semantic fidelity" from a communication perspective has not yet been established, resulting in a lack of theoretical support for semantic consistency optimization. Furthermore, transmission robustness and compression efficiency have not been synergistically optimized. Existing semantic communication systems typically encode and transmit semantic information as a whole without differentiated protection for information at different semantic levels. Under conditions of low SNR or extremely short code lengths, if critical semantic information is treated the same as auxiliary information, errors can easily lead to overall semantic collapse. Moreover, traditional channel coding lacks adaptability in semantic layer applications and fails to incorporate adaptive protection based on semantic importance. In addition, while pure semantic communication reduces transmission load, the high uncertainty in the mapping from high-level semantics to pixel space can easily lead to structural chaos and color distortion in the generated images. Summary of the Invention

[0005] Therefore, it is necessary to provide a semantically driven image reconstruction method and apparatus based on edge and color assistance that can balance semantic accuracy, structural fidelity and color consistency at extremely low bit rates, in order to address the above-mentioned technical problems.

[0006] A semantically driven image reconstruction method based on edge and color assistance, the method comprising:

[0007] The transmitting end performs three parallel feature extractions on the acquired raw image: semantic, edge, and color features, and encodes them separately, outputting semantic encoding sequence, edge encoding sequence, and color encoding sequence;

[0008] Calculate the bit lengths of the semantic encoding sequence, the edge encoding sequence, and the color encoding sequence, and output the header information;

[0009] Based on the header information, the semantic coding sequence, the edge coding sequence, and the color coding sequence are fused to obtain a fused coding sequence; the fused coding sequence is then channel-coded to output a channel-coded sequence.

[0010] The receiving end acquires the channel coding sequence, decodes the channel coding sequence, and outputs the decoded sequence;

[0011] Based on the header information, the decoded sequence is subjected to semantic, edge, and color recovery and decoding to obtain semantic decoding information, edge decoding information, and color decoding information.

[0012] A conditional diffusion model is constructed, and the semantic decoding information, the edge decoding information, and the color decoding information are input into the conditional diffusion model for processing, and the reconstructed image is output.

[0013] On the other hand, a semantically driven image reconstruction apparatus based on edge and color assistance is also provided, comprising:

[0014] The encoding module is used by the transmitter to extract semantic, edge, and color features in parallel from the acquired raw image and encode them separately, outputting semantic encoding sequence, edge encoding sequence, and color encoding sequence;

[0015] The header information output module is used to calculate the bit lengths of the semantic encoding sequence, the edge encoding sequence, and the color encoding sequence, and output the header information;

[0016] The channel coding sequence output module is used to fuse the semantic coding sequence, the edge coding sequence, and the color coding sequence according to the header information to obtain a fused coding sequence; and to perform channel coding on the fused coding sequence to output the channel coding sequence.

[0017] The sequence decoding module is used by the receiving end to acquire the channel-coded sequence, decode the channel-coded sequence, and output the decoded sequence.

[0018] The feature decoding module is used to perform semantic, edge, and color recovery and decoding on the decoded sequence based on the header information to obtain semantic decoding information, edge decoding information, and color decoding information.

[0019] The image reconstruction module is used to construct a conditional diffusion model, input the semantic decoding information, the edge decoding information, and the color decoding information into the conditional diffusion model for processing, and output the reconstructed image.

[0020] Compared with existing technologies, the semantically driven image reconstruction method and apparatus based on edge and color assistance provided by this invention have the following advantages:

[0021] 1. By using semantic information as the core, edge information as structural constraints, and color information as style guidance, this method solves the problems of structural chaos and color distortion in pure semantic reconstruction, and achieves image restoration with accurate semantics, clear structure, and harmonious color, thereby improving the quality of image reconstruction and semantic fidelity.

[0022] 2. By replacing traditional pixel-level transmission with semantic layer transmission, the required information rate is reduced through semantic abstraction, significantly reducing the amount of data transmitted. This breaks through the limits of traditional rate-distortion theory, greatly optimizes compression efficiency, reduces transmission load, and adapts to resource-constrained scenarios. Simultaneously, by incorporating encoding header information, channel errors only affect local data, reducing overall transmission risk, improving system anti-interference capabilities, and enhancing channel transmission robustness.

[0023] 3. The conditional diffusion model injects semantic, edge, and color information as triple conditions into the network, which can flexibly coordinate the three features and output the best reconstruction effect in high, medium and low signal-to-noise ratio environments, adapting to complex and ever-changing communication channels. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the semantically driven image reconstruction method based on edge and color assistance in Example 1.

[0026] Figure 2 This is a flowchart of the semantically driven image reconstruction method based on edge and color assistance in Example 1;

[0027] Figure 3 This is a diagram illustrating the mapping process between the syntactic information set and the semantic information set in Example 1;

[0028] Figure 4 These are comparison images of different schemes obtained under different signal-to-noise ratio conditions in Example 1;

[0029] Figure 5 This is a comparison chart of the SSIM index evaluation of images obtained under different schemes in Example 1;

[0030] Figure 6 This is a comparison chart of the image FID index evaluation obtained under different schemes in Example 1;

[0031] Figure 7 This is a comparison chart of the SA index evaluation of images obtained under different schemes in Example 1;

[0032] Figure 8 This is a structural block diagram of the semantically driven image reconstruction device based on edge and color assistance in Example 2;

[0033] Figure 9 This is a diagram of the internal structure of the computer device in Example 3.

[0034] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0036] It should be noted that in this invention, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0037] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0038] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0039] Example 1

[0040] This embodiment discloses a semantically driven image reconstruction method based on edge and color assistance, aiming to solve the problems of poor image restoration quality and low semantic information fidelity in existing image compression and transmission technologies under low bandwidth and high noise communication environments. Traditional pixel-based compression methods (such as JPEG and WebP) lose a large amount of high-frequency details at extremely low bitrates, resulting in semantic distortion. While pure semantic communication can significantly reduce transmission load, the high uncertainty (i.e., "mapping entropy") in the mapping from high-level semantics to pixel space easily leads to chaotic image structure and color distortion. To address this, this invention proposes a three-element collaborative reconstruction mechanism: using semantic information as the core driver, edge information as the structural constraint, and color information as the style guide, a hierarchical progressive image reconstruction framework is constructed. This framework introduces the concept of mapping entropy at the theoretical level to quantify the information loss during semantic abstraction, concluding that semantic mapping can further reduce the required transmission information rate; at the system level, a joint encoding / decoding architecture of semantics, edge, and color is designed; at the algorithm level, a conditional diffusion model is used to achieve high-fidelity reconstruction; and at the evaluation level, a three-dimensional evaluation system for semantic structure distribution is established.

[0041] like Figure 1 As shown, the semantically driven image reconstruction method based on edge and color assistance provided in this embodiment includes the following steps:

[0042] Step 201: The transmitting end performs three parallel feature extractions (semantic, edge, and color) on the acquired original image and encodes them respectively, outputting semantic encoding sequence, edge encoding sequence, and color encoding sequence.

[0043] Step 202: Calculate the bit lengths of the semantic coding sequence, edge coding sequence, and color coding sequence, and output the header information.

[0044] Step 203: Based on the header information, fuse the semantic coding sequence, edge coding sequence, and color coding sequence to obtain a fused coding sequence; perform channel coding on the fused coding sequence to output the channel coding sequence.

[0045] Step 204: The receiving end acquires the channel coding sequence, decodes the channel coding sequence, and outputs the decoded sequence.

[0046] Step 205: Based on the header information, perform semantic, edge, and color restoration and decoding on the decoded sequence to obtain semantic decoding information, edge decoding information, and color decoding information.

[0047] Step 206: Construct a conditional diffusion model, input semantic decoding information, edge decoding information, and color decoding information into the conditional diffusion model for processing, and output the reconstructed image.

[0048] The flowchart of the image reconstruction method provided by this invention is as follows: Figure 2As shown, the image transmission and reconstruction system consists of a transmitting end (semantic, edge, and color joint coding module) and a receiving end (semantic, edge, and color joint decoding and reconstruction module), forming an end-to-end semantic communication closed loop. Its workflow not only includes information extraction and reconstruction but also integrates three core functions: information compression, channel robustness enhancement, and generation control. This represents a semantic generation function; This represents the edge-assisted semantic reconstruction function. Indicates semantic image generation, This represents an edge-assisted semantic reconstruction image.

[0049] In the specific implementation of step 201, the transmitting end first processes the acquired raw image. Standardization preprocessing includes: setting image specifications, resizing the original image to any size, where... (Image height and width, 3 for RGB channels); color space conversion, preserving RGB space for compatibility with downstream color encoding, but internal semantic / edge extraction can be converted to YUV or Lab space to improve perceptual consistency; boundary filling, using reflection filling to avoid edge artifacts. This invention ensures consistency of input for subsequent modules through standardized preprocessing, while providing a unified processing flow for images of different resolutions.

[0050] After preprocessing, a three-way parallel feature extraction architecture is used to extract the semantic, edge, and color information of the original image, avoiding the accumulation of errors introduced by serial processing.

[0051] In terms of semantic information extraction: Semantic information is extracted from the original image using a panoramic segmentation network, outputting semantic information. The panoramic segmentation network uses a Feature Pyramid Network (FPN) as its backbone, which employs a standard network with multiple spatial resolution features and adds a top-down lightweight path with lateral connections. It is understood that images are typically divided into countable object classes (Thing class, e.g., cars, people) and uncountable region classes (Stuff class, e.g., sky, grass), and it is difficult to recognize all content using only a single neural network scheme. Therefore, the panoramic segmentation network in this embodiment extracts semantic information through instance branches and semantic branches.

[0052] Specifically, in the instance branch, Mask R-CNN (Mask Region-based CNN) with FPN as the baseline is used to detect and label Thing-class objects using bounding boxes, outputting class labels, bounding box offsets, and instance masks. Mask R-CNN includes a ResNet101 backbone network; FPN multi-scale feature pyramid; RPN to generate candidate boxes; RoIAlign to extract instance features; a classification head, a bounding box regression head, and a mask head. The specific structure is a conventional technique and will not be elaborated here.

[0053] In the semantic branch, using FPN as the baseline, the information from each layer of FPN is merged into one for output. Different layer sampling methods are used to assign a class label and instance ID to each pixel at the original image resolution. Pixels with the same label and instance ID belong to the same object. For the "stuff" class label, the instance ID is ignored. When using FPN as the baseline, based on the features of the last layer of FPN, four convolutional layers plus upsampling are applied to output a pixel-level semantic map.

[0054] For example, semantic class tag sets Includes the Thing collection and Stuff set ,Right now Assume the original image contains M class labels and N instance objects. For the class label with ID m, Let n be the instance with number n, where .when At that time, multiple instance IDs will correspond to the same class tag; when During segmentation, class labels are retained while instance IDs are ignored. It's worth noting that during segmentation, each pixel within each instance corresponds to a class label.

[0055] The following describes how to classify each pixel in the image using class labels: Let Original image The pixel coordinates in the image, where, . Represents the original image medium pixel Class tags, Represents pixels For the instance with ID n in the class tag with ID m, then:

[0056] ;

[0057] Through the above partitioning process, the class to which each pixel belongs can be determined, and finally a pixel semantic map is obtained.

[0058] Furthermore, the panoptic segmentation network needs to be trained. During training, the dataset used is COCO Panoptic+ADE20K; the loss function of the panoptic segmentation network is: ,in, , , , Indicates the weighting coefficient; Indicates classification loss; This represents the bounding box regression loss; Indicates instance mask loss; This represents the semantic segmentation loss. The optimizer employs... The trained panoptic segmentation network ultimately achieves panoptic recognition and pixel-level region segmentation of all Thing and Stuff classes in the original image, outputting semantic information, which consists of text describing the image instance. The extracted expression is:

[0059] ;

[0060] In the formula, Represents semantic information; This represents a semantic extraction function.

[0061] Furthermore, since panoptic segmentation combines instance segmentation and semantic segmentation, different instances of the same label will be distinguished. For example, the semantic information after panoptic segmentation is... To further reduce the amount of information transmitted, the extracted identical and synonymous tags will be fused to obtain unified semantic information. The expression is as follows:

[0062] ;

[0063] In the formula, Representing unified semantic information; This represents a semantic fusion function; It represents semantic information.

[0064] The result of the merger This involves merging instances with the same or synonymous tags to obtain the merged information. .

[0065] Regarding edge information extraction: Edge detection algorithms, such as the adaptive threshold Canny operator, are used to extract edge information from the original image. This accurately detects real edge information, and the edge information is denser in areas with rich details, resulting in more accurate image reconstruction. Specifically, the image gradient magnitude is first calculated. , express Directional gradient (horizontal gradient). express The directional gradient (vertical gradient) is calculated; then, a gradient histogram is plotted, and the 5th percentile is used as the low threshold and the 20th percentile as the high threshold; finally, a double-threshold hysteresis connection is applied. Morphological closing operations are then performed on the edge information. Connect the broken edges; remove short line segments (noise) with a length <10 pixels, and output a single-channel binary edge map, where 1 represents an edge. Due to the high sparsity of the edge map (<5% of pixels are 1), the RLC compression ratio can reach over 10:1, which can significantly reduce the amount of data transmitted for edge features, reduce the channel transmission burden, reduce the impact of bit errors on edge features, and provide efficient transmission assurance for preserving key structural constraints in subsequent image reconstruction. The edge information is a binary image, and the extracted expression is:

[0066] ;

[0067] In the formula, Represents edge information; This represents the edge extraction function.

[0068] Regarding color information extraction: To reduce data transmission, a downsampling + interpolation strategy is employed for color information extraction. Specifically, the original image is downsampled using a downsampling method. Downsampling to , This represents the downsampling rate, also known as the scaling factor. After obtaining the color patch, nearest neighbor interpolation upsampling is used to ensure that the final color comes from the original image, without adding new color information or a grayscale-to-color conversion process, and outputs the color information. The color information is presented as a mosaic-like color map. Preferably, the downsampling method uses bicubic interpolation to retain color transitions; the upsampling factor is consistent with the downsampling factor. The expression for color information extraction is:

[0069] ;

[0070] In the formula, Represents color information; This represents the color extraction function.

[0071] It is understandable that the overall color tone style is preserved and high-frequency color details are ignored during the entire color information extraction process, so as to achieve style guidance rather than pixel copying.

[0072] After feature extraction, the three information streams are encoded separately to further reduce the amount of data transmitted. Specifically, semantic information is fused using synonyms to obtain unified semantic information, which is then Huffman encoded to output a semantically encoded sequence. Edge information is run-length encoded to output an edge encoded sequence. Color information is RGB encoded to output a color encoded sequence.

[0073] Specifically, Huffman coding is performed on the unified semantic information. First, all unified semantic information is traversed, treating each character as a leaf node with its frequency as the weight. Then, an optimal binary tree is constructed to generate variable-length prefix codes, outputting the semantically encoded sequence, expressed as:

[0074] ;

[0075] In the formula, Represents a semantically encoded sequence; This represents the Huffman coding function; It represents unified semantic information.

[0076] Run-length encoding (RLC) is applied to the edge information. RLC is a simple lossless data compression method that achieves compression by replacing consecutive character sequences with combinations of the character and its frequency. Based on the binarization characteristics of images, black blocks can be considered as symbols "0" and white blocks as symbols "1," alternating between "0" and "1." Therefore, by simply counting the number of consecutive "0"s and "1"s to form a run-length sequence, lossless transmission can be achieved, outputting the edge-coded sequence. The expression is:

[0077] ;

[0078] In the formula, Represents the edge-coded sequence; This represents the run-length encoding function; Represents edge information.

[0079] The color information is encoded using RGB. For the obtained color image, since each color patch is of uniform size and arranged regularly, each color patch is represented by 24 binary bits. The binary bits of the color of each patch are obtained in a left-to-right, top-to-bottom order, and the color encoding sequence is output as follows:

[0080] ;

[0081] In the formula, Represents a color coding sequence; Represents the RGB encoding function; It indicates color information.

[0082] This embodiment uses three-way parallel extraction and targeted encoding to significantly reduce the amount of data transmitted while preserving key features, laying the foundation for subsequent high-fidelity reconstruction.

[0083] In the specific implementation of step 202, the transmitting end statistically analyzes the semantic coding sequence respectively. Edge-coded sequences Color coding sequence The bit lengths after source coding are denoted as follows: , , Generate header information based on the calculated bit length. This header information is used to record the length information of the three encoded sequences. After the three data streams are combined, a header is added to form a unified transport stream.

[0084] This embodiment uses header information, eliminating the need for tags and preventing ambiguity; parsing is simple, requiring only the length and corresponding bit count to complete; it is also compatible with any encoded content; channel errors only affect a local area, with only length field errors causing overall failure, thus improving transmission stability.

[0085] In the specific implementation of step 203, the semantic coding sequence, edge coding sequence, and color coding sequence are fused according to the bit length in the header information to obtain the fused coding sequence, the expression of which is:

[0086] ;

[0087] In the formula, Indicates a fused encoded sequence; Indicates header information; Represents a semantically encoded sequence; Represents the edge-coded sequence; This represents the color coding sequence.

[0088] After fusion, channel coding is performed on the fused coded sequence using CA-Polar code (CRC-Aided Polar Code). Its basic principle is to utilize the polarization effect to decompose the channel into reliable and unreliable sub-channels, transmitting information bits only on the reliable sub-channels. A 24-bit CRC checksum is appended before or after the information bits for SCL decoding path selection. The code length can be extended to 2048 bits, the information bit rate is 0.5, and the PW (Polarization Weight) algorithm is used for frozen bit selection. The expression for channel coding using CA-Polar code is as follows:

[0089] ;

[0090] In the formula, Represents the channel-coded sequence; Represents the channel coding function; This indicates a fused coded sequence.

[0091] It also includes mapping semantically encoded sequences to more reliable polarization subchannels through a non-uniform protection (UEP) strategy. The non-uniform protection (UEP) strategy is as follows: for semantically encoded sequences... Assign a higher code rate or a stronger CRC check to give higher priority to semantic information.

[0092] This step, through efficient fusion and channel coding, prioritizes the protection of key semantic information and improves transmission robustness in low signal-to-noise ratio environments.

[0093] In the specific implementation of step 204, the receiving end obtains the channel coding sequence sent by the transmitting end through the communication channel and decodes the channel coding sequence using the CA-SCL (CRC-Aided Successive Cancellation List) decoding algorithm.

[0094] Specifically, multiple candidate paths are retained during SCL decoding, and the single, legal, and most probable path is ultimately selected as the output through CRC check. CA-SCL decoding effectively corrects burst errors and improves error plateau performance; moreover, its regular structure makes it easy to implement in hardware, ensuring that the receiver accurately obtains the transmitted information. This applies to the channel coding sequence obtained after encoding. After passing through the channel, the receiver receives a signal containing channel noise. Noisy sequences Finally, through channel decoding, its expression is as follows:

[0095] ;

[0096] In the formula, This represents the decoded sequence; This represents the channel decoding function.

[0097] In the specific implementation of step 205, the receiving end determines the header information. Bit length in , , From the decoded sequence Separate semantic binary sequences from Edge binary sequence Color binary sequence .

[0098] Source decoding is performed on the three binary sequences to obtain semantic decoding information. Edge decoding information Color decoding information This prepares the image for generation. Specifically, it involves semantic binary sequences. Perform Huffman decoding to recover the semantic decoded information. The expression is:

[0099] ;

[0100] In the formula, This represents the Huffman decoding function.

[0101] For edge binary sequences Perform run-length decoding to reconstruct the edge decoding information. The expression is;

[0102] ;

[0103] In the formula, This represents the run-length decoding function.

[0104] RGB decoding of the color binary sequence yields the color decoding features, expressed as:

[0105] ;

[0106] In the formula, This represents the RGB decoding function.

[0107] This step ensures accurate recovery of the three-way features through precise separation and decoding, providing high-quality input for subsequent image reconstruction.

[0108] In the specific implementation of step 206, semantic information can preserve the essential content of the original image, while edge information and color information can, to some extent, compensate for the uncertainty brought about by the mapping process. The conditional diffusion model under conditional control uses edge information and color information as conditional inputs to gradually guide the generation of graphics based on semantic information.

[0109] The conditional diffusion model is based on Stable Diffusion v1.5, whose UNet is used in the latent space. Operationally and computationally efficient. A triple conditional injection method is used to input three channels of decoded information into the model, including semantic decoded information. Converted to text description, then embedded using CLIP Text Encoder. UNet is injected with CrossAttention; edge decoding information is then processed. downsampling to As an additional channel, it is concatenated to the input features of each layer of UNet; the color decoding information is then processed. downsampling to Similarly, they are spliced ​​together as conditional channels.

[0110] The model is trained using the [semantic, edge, color, original image] quadruple as training data (dataset is LAION400M+ COCO). The model is trained using a loss function, the expression of which is:

[0111] ;

[0112] In the formula, This represents the training objective function for the entire model; the smaller the better. Indicates the original image Noise addition progress, i.e., time step Real noise The mathematical expectation; Represents the original image; express Timing-based noise latent representation; This represents the neural network's prediction of additive Gaussian noise; This represents the neural network parameters used for backpropagation updates; This indicates embedded semantic decoding information; Represents edge information; It indicates color information.

[0113] Neural network parameters are used during training. Freezing strategy, backbone UNet parameters Completely freeze to avoid catastrophic amnesia. New edge / color conditional convolutional layer parameters. Trainable. The CrossAttention layer is fine-tunable (LoRA adapter optional); it also employs conditional dropout: randomly masking edge / color conditions (probability 0.1) to improve model robustness; multi-scale supervision: adding auxiliary loss to intermediate layers of UNet; gradient clipping: To prevent explosion.

[0114] The three decoded features are input into the trained conditional diffusion model for processing, and the reconstructed image is output as follows:

[0115] ;

[0116] Due to semantic decoding information Edge decoding information and color decoding information The three types of input information will produce different results in the generated image depending on the degree of their influence, except for the semantic information influence factor. and edge information influence factor In addition, color information influencing factors are also introduced. Therefore, the final equation is:

[0117] ;

[0118] In the formula, Represents the reconstructed image; This represents a color and edge-assisted semantic reconstruction function; Indicates the semantic information influence factor; Indicates the influence factor of marginal information; Indicates the color information influencing factor; Represents semantic decoding information; Indicates edge decoding information; This indicates color decoding information.

[0119] Furthermore, since color information is prone to significant deviations, a dynamic adjustment mechanism is employed to enhance the system's robustness under adverse channel conditions. Specifically, the influencing factor is dynamically adjusted based on the signal-to-noise ratio (SNR) of the communication channel. The dynamic adjustment method is as follows:

[0120] When the signal-to-noise ratio is high (>15dB), the semantic information influence factor, edge information influence factor, and color information influence factor are set to full weight, for example... .

[0121] When the signal-to-noise ratio is medium (5~15dB), the semantic information influence factor and the edge information influence factor are set to full weight, while the weight of the color information influence factor is reduced. For example, .

[0122] When the signal-to-noise ratio is low (<5dB), the semantic information influence factor and the edge information influence factor are set to full weight, and the color information influence factor is set to 0 weight. For example, To avoid misleading color schemes.

[0123] This step, through a conditional diffusion model and a dynamic adjustment strategy, enables semantically accurate, structurally clear, and color-coordinated image reconstruction under different signal-to-noise ratio environments.

[0124] After image reconstruction, a multi-dimensional evaluation system is used to assess the quality of the reconstructed image. Traditional mIoU cannot distinguish between "multiple detections" and "missed detections." This embodiment proposes Semantic Accuracy (SA) to evaluate semantic accuracy, expressed as:

[0125] ;

[0126] In the formula, Indicates semantic accuracy; Represents the number of semantic regions that were correctly detected. ; This represents the number of semantic regions in the original image; This represents the number of semantic regions in the generated graph.

[0127] This indicator also penalizes missed detections. With multiple tests .

[0128] Furthermore, calculating the SSIM index in the Y channel to assess structural similarity is more consistent with human visual perception; the expression is:

[0129] ;

[0130] In the formula, , , These represent the weighting parameters used to adjust the relative importance of brightness, contrast, and structure, respectively. Represents the reconstructed image; Represents the original image; This represents a brightness comparison function; This represents a contrast comparison function; This represents a structure comparison function.

[0131] Let the mean values ​​within the local windows of the two images be respectively , The standard deviation is , covariance is The brightness comparison function is then expressed as: The contrast comparison function is expressed as: The structure comparison function is expressed as: .in, It is a small constant introduced to avoid the denominator being zero, usually .

[0132] Inceptionv3 is used to extract 2048-dimensional features, and the FID index is calculated to evaluate the similarity of image feature distribution. The expression is:

[0133] ;

[0134] In the formula, The mean vector representing the feature vectors of an image; Represents the trace of a matrix; The covariance matrix of image feature vectors; This represents an image reconstructed using edge and color-assisted semantics.

[0135] In one embodiment, by introducing the concept of mapping entropy, the information loss in the semantic abstraction process is quantified, demonstrating that semantic mapping can further reduce the required information rate.

[0136] Traditional information rate distortion function The description is as follows: within the allowable average distortion not exceeding Under the premise that the information transmission rate of the information source can be compressed to the minimum value, that is:

[0137] ;

[0138] In the formula, Represents the original source random variable; Represents the decoded (or reconstructed) random variable; This represents average mutual information at the syntactic level. Represents the coding / channel transition probability, i.e., given the original symbol The reconstructed symbol is The conditional probability; It represents the distortion metric, a non-negative function used for quantization. express The "error" or "distortion" at that time; Indicates average distortion; Indicates the degree of distortion.

[0139] In semantic communication, syntactic information refers to the specific form of expression (e.g., different sentences expressing the same meaning), while semantic information refers to the essential meaning of the expression. For example... Figure 3 As shown, multiple syntactic information can be mapped to the same semantic information, and the syntactic information set is defined as follows: ,in, ,and . Refers to the first The semantic information mapped The Middle One grammatical information, Information set The amount of syntactic information. The semantic information set is... ,in, , That is, one semantic piece of information may correspond to one or more syntactic pieces of information with the same meaning. When syntactic information The probability is , The probability is At that time, semantic information The probability is .

[0140] The probability of semantic information is the sum of the probabilities of all its corresponding syntactic information, expressed as:

[0141] ;

[0142] Semantic entropy The expression is:

[0143] ;

[0144] In the formula, The probability representing semantic information; Represents the probability of syntactic information; Representation of syntactic information set Syntax information in; This indicates the number of syntactic information sets, corresponding to the number of semantic information sets. This represents the information entropy of traditional syntax.

[0145] It can be seen that semantic entropy is no greater than syntactic entropy, and the difference comes from the information compression brought about by semantic abstraction.

[0146] Furthermore,

[0147] Mapping entropy is the uncertainty eliminated in the grammar-to-semantic mapping, expressed as:

[0148] ;

[0149] Inverse mapping entropy is the uncertainty eliminated in the semantic-to-syntactic demapping, expressed as:

[0150] ;

[0151] In the formula, Represents the defined mapping entropy; This represents the entropy of the sending end's syntax information; This represents the semantic information entropy at the sending end; This represents the defined solution mapping entropy; This represents the entropy of the receiving end's syntax information; This represents the semantic information entropy at the receiving end.

[0152] Consider semantic mapping With Remapping The semantic average mutual information is defined as:

[0153] ;

[0154] In the formula, Indicates semantic average mutual information; This represents the joint entropy of the sent syntax information and the received syntax information.

[0155] Using joint entropy relationship We can obtain:

[0156] ;

[0157] In the formula, This represents the average mutual information between the sent and received syntax information.

[0158] Therefore, the semantic rate-distortion function is defined as:

[0159] ;

[0160] This leads to the core conclusion:

[0161] ;

[0162] That is, semantic mapping can further reduce the required information rate of transmission.

[0163] In one embodiment, the superiority of the method proposed in this invention is verified. The test dataset is the COCO dataset (containing rich semantic scenes of the Thing class (people, vehicles, traffic signs, etc.) and the Stuff class (sky, grass, roads, etc.). 500 RGB images with a resolution of 1024×1024 are randomly selected as test samples, covering typical scenes such as daily life, traffic, and nature, to ensure the universality of the verification.

[0164] like Figure 4 As shown, to simulate a low signal-to-noise ratio (SNR) environment in resource-constrained scenarios such as satellite communication and low-altitude communication, the channel SNR is set to: , , , , .

[0165] The comparison benchmarks include JPEG (basic pixel compression), JPEG2000 (wavelet transform compression), and WebP (modern lossy / lossless compression). This invention sets two sets of test variables, Ours1 and Ours2. The condition for the result obtained by Ours1 is that the semantic information, edge information, and color information are affected by channel noise to the same extent during transmission. Ours2 is the image restoration result obtained under the condition that the semantic information is not lost while the edge information and color information are affected by the decrease of SNR, which ensures the comprehensiveness of the comparison.

[0166] The evaluation metrics employ a multi-dimensional indicator system, including the SSIM, FID, and SA metrics. The calculation formulas for each metric have been previously provided. Specifically, the SSIM metric contains three weighting parameters. They are set to 1, 1, and 1 respectively, meaning that the weights of brightness, contrast, and structural information are consistent.

[0167] like Figure 5 As shown, this is a comparison chart of the SSIM index evaluation of images obtained under different schemes. In terms of SSIM index, the method proposed in this invention shows better performance in the low SNR region, while in the high SNR region, the SSIM index of the two schemes proposed in this invention is lower than that of the three traditional schemes.

[0168] like Figure 6As shown, the image FID index evaluation comparison chart obtained under different schemes is presented. In terms of FID index, the two schemes proposed in this invention show better performance in the low SNR region. In the high SNR region, they still have a significant advantage over the traditional JPEG2000 and WebP schemes, and their performance is basically consistent with the JPEG scheme. This shows that the image obtained by the method proposed in this invention can be comparable to the traditional scheme in terms of similarity to the original image, while greatly reducing the amount of information that needs to be transmitted.

[0169] The traditional approach differs from the proposed approach. Below, the accuracy of the same panoramic model in recognizing semantic information in images is also an important factor in evaluating the quality of a solution. For example... Figure 7 As shown, the Ours2 scheme, which prioritizes preserving semantic information in low SNR regions, exhibits a significant advantage overall, accurately reproducing the feature information contained in the original image. The Ours1 scheme also has certain advantages over traditional schemes. Among traditional schemes, the JPEG scheme achieves higher semantic information recognition accuracy compared to JPEG2000 and WebP schemes. This is because JPEG compression results in less image loss, leading to a clearer image compared to JPEG2000 and WebP compression, thus resulting in more accurate semantic recognition. The advantages of the two schemes proposed in this invention also lie in the high image clarity generated using the diffusion model, giving them a significant advantage in semantic recognition.

[0170] In summary, the method proposed in this invention not only maintains high image quality and semantic information accuracy in the field of image compression and reconstruction, but also effectively reduces data transmission volume, demonstrating significant application prospects and technological advantages. Achieving semantically correct, structurally clear, and color-natural image reconstruction at extremely low bit rates provides crucial technical support for bandwidth- and energy-constrained quality-critical scenarios such as satellite remote sensing and IoT vision.

[0171] It should be understood that, although this embodiment Figure 1 The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0172] Example 2

[0173] Based on the semantically driven image reconstruction method based on edge and color assistance in Embodiment 1, this embodiment discloses a semantically driven image reconstruction device based on edge and color assistance, such as... Figure 8 As shown, the semantically driven image reconstruction device based on edge and color assistance includes: an encoding module 401, a header information output module 402, a channel coding sequence output module 403, a sequence decoding module 404, a feature decoding module 405, and a reconstructed image module 406, wherein:

[0174] The encoding module 401 is used by the transmitter to extract semantic, edge, and color features in parallel from the acquired raw image and encode them separately, outputting a semantic encoding sequence, an edge encoding sequence, and a color encoding sequence;

[0175] The header information output module 402 is used to calculate the bit length of the semantic encoding sequence, the edge encoding sequence and the color encoding sequence, and output header information.

[0176] The channel coding sequence output module 403 is used to fuse the semantic coding sequence, the edge coding sequence and the color coding sequence according to the header information to obtain a fused coding sequence; and to perform channel coding on the fused coding sequence to output the channel coding sequence.

[0177] The sequence decoding module 404 is used by the receiving end to acquire the channel coding sequence, decode the channel coding sequence, and output the decoded sequence.

[0178] The feature decoding module 405 is used to perform semantic, edge, and color recovery and decoding on the decoded sequence based on the header information to obtain semantic decoding information, edge decoding information, and color decoding information.

[0179] The image reconstruction module 406 is used to construct a conditional diffusion model, input the semantic decoding information, the edge decoding information, and the color decoding information into the conditional diffusion model for processing, and output the reconstructed image.

[0180] In this embodiment, the specific working process and working principle of the encoding module 401, header information output module 402, channel coding sequence output module 403, sequence decoding module 404, feature decoding module 405, and reconstructed image module 406 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above unit modules.

[0181] Example 3

[0182] like Figure 9 The diagram illustrates a terminal device disclosed in this embodiment, comprising a transmitter, a receiver, a memory, and a processor. The transmitter transmits instructions and data, the receiver receives instructions and data, the memory stores computer-executed instructions, and the processor executes the computer-executed instructions stored in the memory to implement the method described in Embodiment 1 above.

[0183] It is important to note that the aforementioned memory can be either standalone or integrated with the processor. When the memory is set up independently, the terminal device also includes a bus for connecting the memory and the processor.

[0184] Example 4

[0185] This embodiment discloses a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method in Embodiment 1 above.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A semantically driven image reconstruction method based on edge and color assistance, characterized in that, The method includes: The transmitting end performs three parallel feature extractions on the acquired raw image: semantic, edge, and color features, and encodes them separately, outputting semantic encoding sequence, edge encoding sequence, and color encoding sequence; Calculate the bit lengths of the semantic encoding sequence, the edge encoding sequence, and the color encoding sequence, and output the header information; Based on the header information, the semantic coding sequence, the edge coding sequence, and the color coding sequence are fused to obtain a fused coding sequence; the fused coding sequence is then channel-coded to output a channel-coded sequence. The receiving end acquires the channel coding sequence, decodes the channel coding sequence, and outputs the decoded sequence; Based on the header information, the decoded sequence is subjected to semantic, edge, and color recovery and decoding to obtain semantic decoding information, edge decoding information, and color decoding information. A conditional diffusion model is constructed, and the semantic decoding information, the edge decoding information, and the color decoding information are input into the conditional diffusion model for processing, and the reconstructed image is output. The transmitting end performs parallel feature extraction (semantic, edge, and color) on the acquired raw image and encodes them separately, outputting semantic encoded sequences, edge encoded sequences, and color encoded sequences, including: The transmitting end acquires the original image, extracts semantic information from the original image using a panoramic segmentation network, and outputs semantic information; it also extracts edge information from the original image using an edge detection algorithm and outputs edge information; and it extracts color information from the original image using a downsampling method and an interpolation strategy and outputs color information. The semantic information is Huffman encoded to output a semantic encoded sequence; the edge information is run-length encoded to output an edge encoded sequence; and the color information is RGB encoded to output a color encoded sequence.

2. The semantically driven image reconstruction method based on edge and color assistance according to claim 1, characterized in that, Before performing Huffman coding on the semantic information, the following steps are also included: The semantic information is fused using synonyms to obtain unified semantic information; the unified semantic information is then Huffman encoded to output a semantic encoding sequence.

3. The semantically driven image reconstruction method based on edge and color assistance according to claim 2, characterized in that, The semantic information is fused using synonyms to obtain unified semantic information, expressed as follows: ； In the formula, Representing unified semantic information; This represents a semantic fusion function; It represents semantic information.

4. The semantically driven image reconstruction method based on edge and color assistance according to any one of claims 1 to 3, characterized in that, Based on the header information, the semantic encoding sequence, the edge encoding sequence, and the color encoding sequence are fused to obtain a fused encoding sequence, expressed as: ； In the formula, Indicates a fused encoded sequence; Indicates header information; Represents a semantically encoded sequence; Represents the edge-coded sequence; This represents the color coding sequence.

5. The semantically driven image reconstruction method based on edge and color assistance according to claim 4, characterized in that, The process of channel coding the fused coding sequence and outputting the channel coding sequence also includes mapping the semantic coding sequence to a reliable sub-channel using a non-uniform protection strategy.

6. The semantically driven image reconstruction method based on edge and color assistance according to any one of claims 1 to 3, characterized in that, The semantic decoding information, the edge decoding information, and the color decoding information are input into the conditional diffusion model for processing, and the reconstructed image is output. The process expression is as follows: ； In the formula, Represents the reconstructed image; This represents a color and edge-assisted semantic reconstruction function; Indicates the semantic information influence factor; Indicates the influence factor of marginal information; Indicates the color information influencing factor; Represents semantic decoding information; Indicates edge decoding information; This indicates color decoding information.

7. The semantically driven image reconstruction method based on edge and color assistance according to claim 6, characterized in that, In the conditional diffusion model, the influencing factors are dynamically adjusted based on the signal-to-noise ratio of the communication channel to output the reconstructed image; the dynamic adjustment method is as follows: When the signal-to-noise ratio is high, the semantic information influence factor, the edge information influence factor, and the color information influence factor are set to full weight; When the signal-to-noise ratio is medium, the semantic information influence factor and the edge information influence factor are set to full weight, and the weight of the color information influence factor is reduced. When the signal-to-noise ratio is low, the semantic information influence factor and the edge information influence factor are set to full weight, and the color information influence factor weight is set to 0.

8. The semantically driven image reconstruction method based on edge and color assistance according to claim 6, characterized in that, The conditional diffusion model is trained using a loss function, the expression of which is: ； In the formula, This represents the training objective function for the entire model; Indicates the original image Time step Real noise The mathematical expectation; Represents the original image; express Timing-based noise latent representation; This represents the neural network's prediction of additive Gaussian noise; This represents the neural network parameters used for backpropagation updates; This indicates embedded semantic decoding information; Represents edge information; It indicates color information.

9. A semantically driven image reconstruction device based on edge and color assistance, characterized in that, The device includes: The encoding module is used by the transmitter to extract semantic, edge, and color features in parallel from the acquired raw image and encode them separately, outputting semantic encoding sequence, edge encoding sequence, and color encoding sequence; The header information output module is used to calculate the bit lengths of the semantic encoding sequence, the edge encoding sequence, and the color encoding sequence, and output the header information; The channel coding sequence output module is used to fuse the semantic coding sequence, the edge coding sequence, and the color coding sequence according to the header information to obtain a fused coding sequence; and to perform channel coding on the fused coding sequence to output the channel coding sequence. The sequence decoding module is used by the receiving end to acquire the channel-coded sequence, decode the channel-coded sequence, and output the decoded sequence. The feature decoding module is used to perform semantic, edge, and color recovery and decoding on the decoded sequence based on the header information to obtain semantic decoding information, edge decoding information, and color decoding information. The image reconstruction module is used to construct a conditional diffusion model, input the semantic decoding information, the edge decoding information, and the color decoding information into the conditional diffusion model for processing, and output the reconstructed image; In the encoding module, the transmitter performs parallel feature extraction (semantic, edge, and color) on the acquired raw image and encodes them separately, outputting semantic encoded sequences, edge encoded sequences, and color encoded sequences, including: The transmitting end acquires the original image, extracts semantic information from the original image using a panoramic segmentation network, and outputs semantic information; it also extracts edge information from the original image using an edge detection algorithm and outputs edge information; and it extracts color information from the original image using a downsampling method and an interpolation strategy and outputs color information. The semantic information is Huffman encoded to output a semantic encoded sequence; the edge information is run-length encoded to output an edge encoded sequence; and the color information is RGB encoded to output a color encoded sequence.

Images