A semantic importance guided image encoding and wireless transmission system and method

Abstract

The application discloses a semantic importance guided image coding and wireless transmission system and method, which generates a semantic importance map by analyzing the importance of pixels in an image to a downstream machine vision task; based on the map and a channel state, a joint optimization model of a source and a channel code rate allocation under a total code rate constraint is constructed to generate a coding quality vector and a channel code rate allocation vector; then, the image is subjected to semantic guided non-uniform compression, and a code stream is subjected to differential channel coding and transmission. The application breaks the traditional source-channel separation architecture, prioritizes the reliable transmission of high semantic value areas under limited communication resources, effectively balances the machine vision task accuracy and human eye perception quality, and significantly improves the operation reliability and efficiency in a human-machine collaborative scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a semantically important guided image coding and wireless transmission system and method. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), 5G communication, and artificial intelligence technologies, complex human-machine collaborative application scenarios such as drone power line inspection, telemedicine, and emergency rescue are becoming increasingly common. In these scenarios, image data not only needs to be transmitted to backend machine vision algorithms for real-time analysis (such as target detection and fault identification), but also needs to be presented to human operators for review or decision-making.

[0003] Traditional image / video coding technologies currently rely primarily on international standards for image and video transmission, such as JPEG, H.264 / AVC, H.265 / HEVC, and H.266 / VVC. These standards are typically based on hybrid coding frameworks, employing techniques such as prediction, transform, quantization, and entropy coding. However, they suffer from the following drawbacks: Traditional coding standards are designed to optimize human visual perception quality (e.g., PSNR, SSIM), aiming to restore pixel details with minimal bitrate. However, they neglect the semantic information of image content and the requirements of downstream machine vision tasks. Under high compression rates due to limited wireless channel bandwidth or network congestion, traditional methods apply uniform, strong compression to images, leading to blurred edges and texture loss of key machine vision targets, severely reducing the recognition accuracy of machine vision algorithms. Furthermore, this indiscriminate coding approach wastes a significant amount of valuable bitrate resources in task-irrelevant background areas.

[0004] Existing wireless image transmission systems strictly adhere to Shannon's separation theorem, designing source coding (application layer compression) and channel coding (physical layer error correction) as two independent modules. The source encoder is responsible for removing redundancy, while the channel encoder is responsible for adding redundancy to combat noise. This separation architecture has the following drawbacks: While near-optimal for long code lengths and static channels, it exhibits a significant "cliff effect" in dynamic, time-varying channels (such as high-speed drone movement) and short code length transmissions. Specifically, when the channel signal-to-noise ratio (SNR) falls below a certain threshold, decoding will completely fail, leading to packet loss or image interruption. More importantly, physical layer transmission treats all data packets equally, failing to perceive whether a data packet carries critical semantic information or redundant background information. This results in the loss of critical target bitstreams due to insufficient error correction capabilities when the channel deteriorates, while irrelevant background bitstreams may be successfully transmitted. This misalignment of resource allocation significantly reduces the system's mission reliability.

[0005] In recent years, the academic community has proposed the Deep Learning-based Joint Source-Channel Coding (Deep JSCC) technique. This technique utilizes an autoencoder to directly map an image to channel input symbols, achieving joint optimization of the source and channel. Although Deep JSCC exhibits good robustness at low signal-to-noise ratios, most current schemes suffer from the following problems: Poor hardware compatibility: Most DeepJSCC solutions output analog symbols or non-standard constellation diagrams, making it difficult to directly adapt to existing standardized digital communication hardware (such as 4G / 5G and WiFi modules). The needs of both humans and machines are difficult to meet simultaneously: most existing solutions focus on a single optimization objective (either optimizing only the accuracy of machine tasks, resulting in low quality of reconstructed images; or optimizing only human vision, where the accuracy of machine tasks cannot be guaranteed). Therefore, under limited communication resources, existing image encoding and transmission solutions cannot simultaneously meet the dual needs of "machine analysis + human review" in human-machine collaborative scenarios; Computational complexity and adaptability are at odds: In order to adapt to dynamic channels, it is often necessary to train multiple models with different signal-to-noise ratios and switch between them. The storage and computational overhead is huge, making it difficult to deploy and process in real time on resource-constrained embedded terminals. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention aims to provide a semantically important guided image encoding and wireless transmission system and method that can adaptively balance machine vision accuracy and human visual perception quality under dynamically changing wireless channels, significantly improving the reliability and efficiency of operations in human-machine collaborative scenarios.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A semantically important guided image coding and wireless transmission system includes a semantic importance analysis module, an adaptive image semantic coding module, a latent vector block module, a source-channel code rate joint allocation module, a code stream grouping module, a channel encoder, and a modulator deployed at the transmitting end, and a demodulator, a channel decoder, and an adaptive image semantic decoding module deployed at the receiving end. The semantic importance analysis module is used to analyze the importance of each pixel in the input original image X to the downstream machine vision task, and generate a pixel-level semantic importance map. ; The source channel code rate joint allocation module receives the semantic importance mapping. The implicit vector block information and channel state information are used to construct a joint optimization model of source and channel code rate under the total code rate constraint, generating a coding quality vector M and a channel code rate allocation vector r, which respectively guide the semantic encoder to perform task-aware semantic coding and the channel encoder to perform semantic-first channel coding. The adaptive image semantic coding module performs non-uniform, task-related compression coding on the original image based on the coding quality vector M. The adaptive image semantic coding module includes a semantic encoder, a latent vector block module, a quantizer, and an entropy encoder. The semantic encoder employs a deep neural network model trained and converged end-to-end. It takes the original image X and the encoded quality vector M as input, extracts image features adapted to the bit rate and downstream tasks, and outputs a latent vector Z. The latent vector block module divides the latent vector Z into blocks of equal size to obtain the latent vectors. The quantizer processes the hidden vectors after block division. Perform scalar quantization to obtain the latent vector. The entropy encoder processes the quantized latent vectors. Entropy encoding is performed to generate the source code stream L; The code stream grouping module generates code stream groups based on the source code stream L output by the entropy encoder and the channel code rate allocation strategy r, combined with the channel coding group size. And related signaling; this signaling includes channel coding rate and source code stream attribution information, used for channel encoding / decoding and receiver entropy decoding; The channel encoder is responsible for grouping the code stream at the transmitting end according to the channel code rate allocation vector r and signaling. Perform channel coding; The modulator is responsible for modulating the channel-coded packet stream into a baseband signal; The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the demodulator to the receiving end; after receiving the signaling and baseband signal, the receiving end parses the signaling and then performs the following processing: The demodulator demodulates the received baseband signal; The channel decoder performs block decoding based on the parsed signaling to obtain the reconstructed source code stream; The adaptive image semantic decoding module includes a semantic decoder and an entropy decoder. The semantic decoder uses a deep neural network model that has been trained and converged end-to-end. The entropy decoder performs entropy decoding on the code stream output by the channel decoding at the receiving end based on the signaling information to reconstruct the latent vector. The semantic decoder completes image reconstruction based on the reconstructed latent vector.

[0008] The source-channel code rate joint allocation module, based on the premise of consistency between the relative positions in the latent vector space and the pixel space, utilizes latent vector block information to map the semantic importance graph. Perform chunking processing, specifically mapping the semantic importance graph. The region is divided into regions aligned with the latent vector block space, and the maximum value within the region is selected as the semantic importance weight of the block. This ensures that pixels and latent vector elements corresponding to the same block share the same semantic weight and coding parameters, thereby outputting the block-based coding quality vector M and the channel code rate allocation vector r.

[0009] The adaptive image semantic decoding module and the adaptive image semantic decoding module are trained using a step-by-step optimization strategy: Step T1, Basic Training: Perform pixel-wise rate-distortion optimization iterative training based on multi-rate coding with coding quality vector control. The optimization objective function of the training process is:

[0010] in, The estimated entropy of the quantized latent vector is given by H and W, where H and W are the height and width of the image, respectively, and HW is the total number of pixels in the image. The weight vector is composed of the mean square error of each pixel. elements in The corresponding elements in the encoded quality vector are processed by a preset monotonically increasing function. Mapped to obtain; Step T2, Task-Aware Training: Based on the convergence of basic training, fix the entropy model parameters, add downstream task-related optimization objective functions, and perform bitrate-distortion-task accuracy optimization training for encoding and decoding, where the task accuracy optimization term is related to the specific downstream task; Step T3, Human Eye Perception Quality Optimization: Freeze the parameters of the semantic encoder and fine-tune only the semantic decoder; introduce a discriminator network and use generative adversarial loss to train the semantic decoder.

[0011] In step T2, the added downstream task-related optimization objective function is: , This refers to the loss function corresponding to the downstream machine vision task. The weight of the task loss item.

[0012] The source-channel code rate joint allocation module constructs a source-channel code rate joint optimization model under the total code rate constraint based on an elite genetic algorithm, generating a coding quality vector M and a channel code rate allocation vector r, as follows: Step 2-1: The source-channel code rate joint allocation module maps the semantic importance graph. Downsampling is performed to the same dimension as the latent vector blocks, and partitioning is performed based on the block information. The maximum value within each partition is taken as the semantic importance value of that latent vector block, thus obtaining the semantic importance vector. Calculate the total bit limit under the current channel conditions using Shannon's formula. ; Step 2-2: Construct the source-channel joint code rate allocation optimization model as follows: , Among them, the image coding quality vector Divided into N regions, where different elements within a region have the same value. , For discrete variables, their set of values ​​is defined as follows: The number of bits in each latent vector block bitstream output by the semantic encoder is The code length of each packet in the channel coding is... Channel code rate allocation vector , The set of values ​​is defined as ;for the bitstream Channel coding is performed, and the number of effective information bits per packet in the resulting channel code stream is: The error probability for each group is , bitstream After channel coding, the bits are The semantic importance of image segmentation lies in the fact that the encoded image blocks must be correctly transmitted to the receiving end to generate revenue. Let $\frac{ ... , where the function Monotone non-decreasing function; Steps 2-3: Use an elite genetic algorithm to solve the source-channel joint code rate allocation optimization model to obtain the image coding quality allocation vector and the channel code rate allocation vector.

[0013] Steps 2-3 are as follows: (1) Individual coding: A chromosome consists of two segments: a source code rate allocation segment and a channel code rate allocation segment, and the two segments are independent of each other, meaning they have different gene sets; (2) Definition of fitness function; The fitness function is constructed based on the semantic importance of each block: , Fit is the expected semantic gain that can be obtained after the image is jointly encoded and transmitted through the source and channel. Based on this fitness function, the optimization objective of the elite genetic algorithm is to maximize Fit while keeping the total transmission bit rate constant. (3) Constraints and optimization of feasible solutions; 1) The total number of bits transmitted does not exceed the channel capacity (Restriction 1): The source channel code rate allocation must also meet the requirement that the total number of bits transmitted through the channel is within the specified range. Less than the channel capacity By adding a penalty to the fitness function, the final fitness function is obtained as follows: ,in, This is a normalization function used to adjust the value of the penalty term.

[0014] 2) Monotonicity of Source Code Rate and Importance for Each Block (Restriction 2): This allocation mechanism, given a fixed total number of bits transmitted, prioritizes allocating higher source code rates to image blocks with high semantic importance and lower channel code rates to their corresponding bitstreams. This ensures that image blocks with high semantic importance have higher coding accuracy and can be transmitted to the receiver with a higher success rate. Therefore, the source coding rate of each block is a non-decreasing function of the block's semantic importance, and the channel code rate is a non-increasing function of the block's semantic importance.

[0015] 3) Optimization of feasible solutions: New individuals are generated randomly, and the new individuals must meet the above constraints; After several generations of propagation, the optimal coding quality vector is output. and the channel code rate allocation vector for each hidden vector block .

[0016] A semantically important guided image coding and wireless transmission method is disclosed. The method employs the semantically important guided image coding and wireless transmission method described above. The method first trains an adaptive image semantic coding module and an adaptive image semantic decoding module using a step-by-step optimization strategy, and then performs the following steps: Step 1: The semantic importance analysis module at the sending end analyzes the importance of each pixel in the input original image X to the downstream machine vision task, and generates a pixel-level semantic importance map. ; Meanwhile, the latent vector segmentation module generates segmentation information based on the downsampling rate of the semantic encoder and the original image size; Step 2: Map the semantic importance graph The module takes block information and channel state as input, performs source and channel code rate joint allocation, runs the source and channel code rate joint allocation operation, and outputs a coding quality vector for source code rate allocation. and the channel code rate allocation vector for each hidden vector block ,in, The elements in the array take values ​​of [0,1] and are divided into N regions; The channel code rate for each block, It is a one-dimensional vector with a length of N; Step 3: Input the original image X and the encoded quality vector M into the semantic encoder to obtain the latent vector Z. Divide the latent vector Z into blocks, and quantize and entropy encode each block to obtain the bitstream of each latent vector block. ; Step 4: The code stream grouping module allocates vectors according to the channel code rate. The code length of the channel coding, for the bit stream The information packets are obtained by grouping them to obtain channel-coded information packets. During the code stream grouping operation, the hidden vector block to which each information group belongs and the channel code rate of the group are recorded. The above recorded information is used as signaling to guide the channel encoder to perform channel coding. Step 5: The channel encoder groups the information bits according to the signaling. Channel coding is performed separately, and the modulator modulates the coded channels into baseband signals. Step 6: The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the modulator to the receiving end; after receiving the signaling and baseband signal, the receiving end performs the following processing: The demodulator at the receiving end demodulates the received signal, the receiving end performs signaling parsing, and the channel decoder performs channel decoding based on the signaling to obtain the image bitstream. Channel decoding: Decoding is performed according to the code rate indicated by the signaling; Image reconstruction: The semantic decoder receives the reconstructed latent vectors and outputs the reconstructed image. .

[0017] In step 2, the joint allocation of source and channel code rates is performed based on the elite genetic algorithm, as follows: Step 2-1: The source-channel code rate joint allocation module maps the semantic importance graph. Downsampled to the same dimension as the latent vector blocks, and partitioned according to the block information, the maximum value within each partition is taken as the semantic importance value of that latent vector block. This yields the semantic importance vector. Calculate the total bit limit under the current channel conditions using Shannon's formula. ; Step 2-2: Construct the source-channel joint code rate allocation optimization model as follows: , Among them, the image coding quality vector Divided into N regions, where different elements within a region have the same value. , For discrete variables, their set of values ​​is defined as follows: The number of bits in each latent vector block bitstream output by the semantic encoder is The code length of each packet in the channel coding is... Channel code rate allocation vector , The set of values ​​is defined as ;for the bitstream Channel coding is performed, and the number of effective information bits per packet in the resulting channel code stream is: The error probability for each group is , bitstream After channel coding, the bits are The semantic importance of image segmentation lies in the fact that the encoded image blocks must be correctly transmitted to the receiving end to generate revenue. Let the revenue from this block be denoted as , and let the encoding quality be positively correlated with the block revenue. The revenue that can be obtained upon receipt can be expressed as . , where the function Monotone non-decreasing function; Steps 2-3: Use an elite genetic algorithm to solve the source-channel joint code rate allocation optimization model to obtain the image coding quality allocation vector and the channel code rate allocation vector.

[0018] Steps 2-3 are as follows: (1) Individual coding: A chromosome consists of two segments: a source code rate allocation segment and a channel code rate allocation segment, and the two segments are independent of each other, meaning they have different gene sets; (2) Definition of fitness function; The fitness function is constructed based on the semantic importance of each block: , Fit is the expected semantic gain that can be obtained after the image is jointly encoded and transmitted through the source and channel. Based on this fitness function, the optimization objective of the elite genetic algorithm is to maximize Fit while keeping the total transmission bit rate constant. (3) Constraints and optimization of feasible solutions; 1) The total number of bits transmitted does not exceed the channel capacity (Restriction 1): The source channel code rate allocation must also meet the requirement that the total number of bits transmitted through the channel is within the specified range. Less than the channel capacity By adding a penalty to the fitness function, the final fitness function is obtained as follows: ,in, This is a normalization function used to adjust the value of the penalty term.

[0019] 2) Monotonicity of Source Code Rate and Importance for Each Block (Restriction 2): This allocation mechanism, given a fixed total number of bits transmitted, prioritizes allocating higher source code rates to image blocks with high semantic importance and lower channel code rates to their corresponding bitstreams. This ensures that image blocks with high semantic importance have higher coding accuracy and can be transmitted to the receiver with a higher success rate. Therefore, the source coding rate of each block is a non-decreasing function of the block's semantic importance, and the channel code rate is a non-increasing function of the block's semantic importance.

[0020] 3) Optimization of feasible solutions: New individuals are generated randomly, and the new individuals must meet the above constraints; After several generations of propagation, the optimal coding quality vector is output. and the channel code rate allocation vector for each hidden vector block .

[0021] The adaptive image semantic decoding module and its training are as follows: Step T1, Basic Training: Perform pixel-wise rate-distortion optimization iterative training based on multi-rate coding controlled by the coding quality vector M. The optimization objective function of the training process is:

[0022] in, The estimated entropy of the quantized latent vector is given by H and W, where H and W are the height and width of the image, respectively, and HW is the total number of pixels in the image. The weight vector is composed of the mean square error of each pixel. elements in The corresponding elements in the encoded quality vector are processed by a preset monotonically increasing function. Mapped to obtain; Step T2, Task-Aware Training: Based on the convergence of basic training, fix the entropy model parameters, add downstream task-related optimization objective functions, and perform bitrate-distortion-task accuracy optimization training for encoding and decoding, where the task accuracy optimization term is related to the specific downstream task; Step T3, Human Eye Perception Quality Optimization: Freeze the parameters of the semantic encoder and fine-tune only the semantic decoder; introduce a discriminator network and use generative adversarial loss to train the semantic decoder.

[0023] By adopting the above scheme, this invention breaks the traditional architecture of separating the source and channel in communication. It extracts pixel-level semantic importance through deep learning and uses this as a priori for source and channel code rate allocation. At the source coding level, it achieves task-aware non-uniform compression, prioritizing the retention of image regions most critical for machine discrimination and human visual verification under limited resources. At the channel coding level, it constructs a joint resource allocation model based on an improved elite genetic algorithm, introducing innovative "monotonicity constraints" and "gene repair strategies" to ensure stronger channel protection for important data blocks. Ultimately, under dynamically changing wireless channels, the system can adaptively balance machine vision accuracy and image reconstruction quality, significantly improving the reliability and efficiency of operations in human-machine collaborative scenarios. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the principle of the present invention; Figure 2 This is a schematic diagram illustrating an application scenario of an embodiment of the present invention. Detailed Implementation

[0025] like Figure 1 As shown, this invention discloses a semantically important guided image coding and wireless transmission system, which includes a semantic importance analysis module, an adaptive image semantic coding module, a latent vector block module, a source-channel code rate joint allocation module, a code stream grouping module, a channel encoder, and a modulator deployed at the transmitting end, and a demodulator, a channel decoder, and an adaptive image semantic decoding module deployed at the receiving end.

[0026] The semantic importance analysis module is used to analyze the importance of each pixel in the input original image X to the downstream machine vision task, and generate a pixel-level semantic importance map. This semantic importance mapping graph The semantic importance of each pixel in the image is characterized, and the original mapping is output to the source-channel code rate joint allocation module to guide the image semantic coding and channel code rate allocation.

[0027] The latent vector segmentation module segments the latent vectors in a planar dimension and establishes an association between each segment and the semantic importance mapping based on the correspondence between the latent space and the pixel space (i.e., relative position consistency). This determines the importance weight of each latent vector segment and provides a basis for the joint allocation of source and channel code rates.

[0028] The latent vector block segmentation module logically divides the latent vectors output by the adaptive image semantic coding module into blocks, generating latent vector block information. This block information includes the number of blocks, their arrangement, and the index of the latent vector elements contained in each block, providing a spatial index basis for subsequent bitrate allocation. Specifically, the latent vector block information is calculated by this module based on the original image size and a preset image block size. It should be noted that there is a fixed numerical mapping relationship between the image block size in physical space and the latent vector block size in latent space (this relationship is determined by the downsampling ratio of the semantic coding network), thus ensuring a one-to-one correspondence between the latent vector blocks and the original image blocks in spatial location.

[0029] Source-channel code rate joint allocation module receive semantic importance mapping diagram Based on latent vector block information and channel state information, a joint optimization model for source and channel bit rates is constructed under the constraint of total bit rate. This model generates a coding quality vector M and a channel bit rate allocation vector r, which respectively guide the semantic encoder to perform task-aware semantic coding and the channel encoder to perform semantic-priority channel coding, thus prioritizing the accuracy of downstream tasks under resource-constrained conditions. To reduce the computational complexity of the joint optimization and adapt to actual hardware processing capabilities, this module, based on the premise of consistency between the relative positions of the latent vector space and the pixel space, utilizes latent vector block information to map semantic importance. Perform chunking processing, specifically mapping the semantic importance graph. The region is divided into regions aligned with the latent vector block space, and the maximum value within the region is selected as the semantic importance weight of the block. This ensures that pixels and latent vector elements corresponding to the same block share the same semantic weight and coding parameters, thereby outputting the block-based coding quality vector M and the channel code rate allocation vector r.

[0030] The adaptive image semantic coding module performs non-uniform, task-related compression coding on the original image based on the coding quality vector M.

[0031] Specifically, the adaptive image semantic coding module includes a semantic encoder, a latent vector block module, a quantizer, and an entropy encoder. The semantic encoder employs a deep neural network model trained and converged end-to-end, capable of adjusting the coding quality and bitrate of different regions based on the coding quality vector M. The deep neural network is not limited to CNNs (Convolutional Neural Networks) and can also be built based on architectures such as Vision Transformer (ViT) and Swing Transformer.

[0032] The semantic encoder takes the original image X and the encoded quality vector M as input, extracts image features adapted to the bit rate and downstream tasks, and outputs a latent vector Z. The latent vector segmentation module divides the latent vector Z into blocks of equal size, obtaining the latent vector... The quantizer processes the hidden vectors after partitioning. Perform scalar quantization to obtain the latent vector. The entropy encoder processes the quantized latent vectors. Entropy encoding is performed to generate the source code stream L.

[0033] The code stream packetization module generates code stream packets based on the source code stream L output by the entropy encoder and the channel code rate allocation strategy r, combined with the channel coding packet size. And related signaling; this signaling includes channel coding rate and source code stream attribution information, which is used for channel coding and decoding and receiver entropy decoding.

[0034] The channel encoder is responsible for packetizing the code stream at the transmitting end according to the code rate allocation strategy r and signaling. Perform channel coding.

[0035] The modulator is responsible for modulating the channel-coded packet stream into a baseband signal.

[0036] The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the demodulator to the receiving end. After receiving the signaling and baseband signal, the receiving end parses the signaling and then performs the following processing: The demodulator demodulates the received baseband signal.

[0037] The channel decoder performs block decoding based on the parsed signaling to obtain the reconstructed source code stream.

[0038] The adaptive image semantic decoding module includes a semantic decoder and an entropy decoder. The semantic decoder employs a deep neural network model trained and converged end-to-end. The entropy decoder performs entropy decoding on the code stream output from the channel decoding at the receiving end based on signaling information, reconstructing the latent vectors. The semantic decoder then completes image reconstruction based on the reconstructed latent vectors.

[0039] This embodiment applies the above system to a human-machine collaborative scenario for unmanned aerial vehicle (UAV) power line inspection. In this scenario, the UAV (transmitter) is equipped with a high-definition camera to capture images of power equipment and transmits the images to a ground control station (receiver) via a wireless channel.

[0040] Machine vision task: The ground station is equipped with a deep learning-based image classification model (such as ResNet-50) to automatically identify whether there are faults in the equipment in the image (such as "damaged insulator", "normal", "foreign object hanging").

[0041] Human-machine collaboration: Ground station operators need to view the reconstructed images to manually verify the machine's judgment results.

[0042] Resource constraints: Wireless transmission bandwidth is limited and channel conditions fluctuate greatly. Priority should be given to ensuring machine classification accuracy while improving the human visual perception experience as much as possible.

[0043] Before the system is actually run, the adaptive image semantic decoding module and the adaptive image semantic decoding module need to be trained. This embodiment uses a step-by-step optimization strategy for training: Step T1 (Basic Training): Perform pixel-wise rate-distortion optimization iterative training based on multi-rate coding controlled by the coding quality vector M. The optimization objective function of the training process is:

[0044] in, The estimated entropy of the quantized latent vector is given by H and W, where H and W are the height and width of the image, respectively, and HW is the total number of pixels in the image. The weight vector is composed of the mean square error of each pixel. elements in The corresponding elements in the encoded quality vector M ( The value range is [0,1], and it is obtained through a preset monotonically increasing function. Mapped to obtain, The larger the value, the greater the corresponding weight, which makes the optimization process more inclined to reduce the error at that position, ultimately enabling the pixel at that position to obtain higher encoding quality.

[0045] Step T2 (Task-Aware Training): Based on the convergence of the basic training, fix the entropy model parameters, add the downstream task-related optimization objective function, and perform bitrate-distortion-task accuracy optimization training for encoding and decoding. The optimization term for task accuracy is related to the specific downstream task.

[0046] In this embodiment, the added downstream task-related optimization objective function is: , This refers to the loss function corresponding to the downstream machine vision task. The weight of the task loss item.

[0047] Step T3 (Human Perception Quality Optimization): Freeze the parameters of the semantic encoder and fine-tune only the semantic decoder. Introduce a discriminator network and use generative adversarial loss to train the semantic decoder to improve the texture realism of the reconstructed image, making the distribution of the reconstructed image closer to that of the natural image.

[0048] In this embodiment, the specific steps of the image encoding and wireless transmission system operation are as follows: Step 1: The semantic importance analysis module at the sending end analyzes the importance of each pixel in the input original image X to the downstream machine vision task, and generates a pixel-level semantic importance map. .

[0049] In this embodiment, the drone acquires the original image. (resolution is) The semantic importance analysis module employs the Grad-CAM (Gradient-weighted Class Activation Mapping) algorithm, utilizing a pre-trained classification model to calculate the target class relative to the input original image. The gradient is used to generate pixel-level heatmaps. Higher heatmap values ​​indicate greater importance of the region to the classification result (e.g., the region where the insulator is located). In addition to using Grad-CAM (Class Activation Mapping), attention mechanisms and saliency detection algorithms can also be used to generate semantic importance maps.

[0050] Simultaneously, the latent vector segmentation module generates segmentation information based on the semantic encoder's downsampling rate and the original image size. For example, if the semantic encoder's downsampling rate is set to 16, the generated latent vector dimension is... The latent vector segmentation module further divides the latent vectors into 16 groups. Small blocks.

[0051] Step 2: Map the semantic importance graph The module takes block information and channel state (signal-to-noise ratio and bandwidth) as input to the source-channel code rate joint allocation module, runs a source-channel code rate joint allocation operation based on an elite genetic algorithm, and outputs a coding quality vector for source code rate allocation. and the channel code rate allocation vector for each hidden vector block , where vector The system is divided into N regions based on the block information. The elements in each region have the same value, and the value range is [0,1]. It is a one-dimensional vector with length N, and vector elements The channel code rate for each block.

[0052] Step 2 is as follows: Step 2-1 (Preprocessing): The source-channel code rate joint allocation module maps the semantic importance graph. Downsampling is performed to the same dimension as the latent vector blocks, and partitioning is performed based on the block information. The maximum value within each partition is taken as the semantic importance value of that latent vector block, thus obtaining the semantic importance vector. , . Calculate the total bit limit under the current channel conditions using Shannon's formula. .

[0053] Step 2-2 (Constructing the Optimization Model): Constructing a joint source-channel code rate allocation optimization model. Image coding quality vector. Divided into N regions, where different elements within a region have the same value. In this embodiment For discrete variables, their set of values ​​is defined as follows: The number of bits for each latent vector block output by the semantic encoder is... The code length of each packet in the channel encoder is... Channel code rate allocation vector , The set of values ​​is defined as ;for the bitstream Channel coding is performed, and the number of effective information bits per packet in the resulting channel code stream is: The error probability for each group is , bitstream After channel coding, the bits are The semantic importance of image segmentation lies in the fact that the encoded image blocks must be correctly transmitted to the receiving end to generate revenue. Let the revenue from this block be denoted as , and let the encoding quality be positively correlated with the block revenue. The revenue that can be obtained upon receipt can be expressed as . , where the function For a monotonically non-decreasing function, the resulting optimization model is as follows: .

[0054] Steps 2-3: The source-channel joint code rate allocation optimization model is solved using an elite genetic algorithm to obtain the image coding quality allocation vector and the channel code rate allocation vector. Details are as follows: (1) Individual coding: A chromosome consists of two segments: a source code rate allocation segment and a channel code rate allocation segment. These two segments are independent of each other, meaning they contain different gene sets. The source code rate allocation is discrete. K The channel code rate allocation level is discrete as follows: V The gene set of the source code rate allocation segment is {0, 1, 2, 3, 4, ...,}. K -1}, corresponding to the source code rate level of each block; the gene set of the channel code rate allocation segment is {0, 1, 2, ..., V -1} corresponds to the channel code rate level of each block of the latent vector.

[0055] Based on the above gene set, a chromosome consists of two segments of length N, corresponding to the source code rate allocation level and the channel code rate allocation level, respectively.

[0056] (2) Definition of fitness function The fitness function is constructed based on the semantic importance of each block: .

[0057] Based on this fitness function, the optimization objective of the genetic algorithm is to maximize Fit while keeping the total transmission bit rate constant. Fit represents the expected semantic gain that the image can obtain after being jointly encoded and transmitted through the source and channel.

[0058] (3) Constraints and optimization of feasible solutions 1) The total number of bits transmitted does not exceed the channel capacity (this is constraint 1): The source channel code rate allocation must also meet the requirement that the total number of bits transmitted through the channel is within the specified range. Less than the channel capacity By adding a penalty to the fitness function, the final fitness function is obtained as follows:

[0059] in, This is a normalization function used to adjust the value of the penalty term.

[0060] 2) Monotonicity of source channel code rate and importance for each block (this is constraint 1): This allocation mechanism, given a fixed total number of bits transmitted, prioritizes allocating higher source code rates to image blocks with high semantic importance and lower channel code rates to their corresponding bitstreams. This ensures that image blocks with high semantic importance have higher coding accuracy and can be transmitted to the receiver with a higher success rate. Therefore, the source coding code rate for each block is a non-decreasing function of the block's semantic importance, and the channel code rate is a non-increasing function of the block's semantic importance.

[0061] 3) Optimization of feasible solutions: New individuals are generated randomly, and the new individuals must meet the above constraints.

[0062] Traditionally, individuals that do not meet the constraints are discarded and new random individuals are generated. However, as chromosome length increases, the space for randomly generated chromosomes becomes excessive, leading to a large number of operations being concentrated on individual initialization. Therefore, this invention proposes the following two strategies: For constraint 1, a penalty mechanism is introduced to eliminate chromosomes exceeding the total code rate; for constraint 2, randomly generated chromosomes that do not meet the monotonicity constraint are not discarded, but their genes are redistributed to satisfy monotonicity. Furthermore, to increase population diversity, in each iteration, the top few individuals with the highest fitness are retained, and new individuals are added to maintain a consistent total population size.

[0063] After several generations of propagation, the optimal coding quality vector is output. and the channel code rate allocation vector for each hidden vector block .

[0064] Step 3: Input the original image X and the encoded quality vector M into the semantic encoder to obtain the latent vector Z. Divide the latent vector Z into blocks, and quantize and entropy encode each block to obtain the bitstream of each latent vector block. .

[0065] In this embodiment, high bitrates are allocated to high-importance regions such as "insulators" in the encoded quality vector M to preserve rich details; while low bitrates are allocated to background regions such as "sky" and "grass".

[0066] Step 4: The code stream grouping module allocates vectors according to the channel code rate. The code length of the channel coding, for the bit stream The information packets are obtained by grouping them to obtain channel-coded information packets. During the code stream grouping process, the hidden vector block to which each information group belongs and the channel code rate of the group are recorded. The above recorded information is used as signaling to guide the channel encoder to perform channel coding.

[0067] Step 5: The channel encoder groups the information bits according to the signaling. Channel coding is performed separately, and the modulator modulates the coded channels into baseband signals.

[0068] In this embodiment, the channel encoder uses a low code rate (such as 1 / 3 LDPC code) to provide strong protection for high-importance packets, and a high code rate (such as 3 / 4 LDPC code) to provide protection for background packets.

[0069] Step 6: The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the modulator to the receiving end. Upon receiving the signaling and baseband signal, the receiving end performs the following processing: The demodulator at the receiving end demodulates the received signal, the receiving end performs signaling parsing, and the channel decoder performs channel decoding based on the signaling to obtain the image bitstream.

[0070] Channel decoding: Decoding is performed according to the code rate indicated by the signaling. Due to strong protection of important areas, even under conditions of fluctuating signal-to-noise ratio, key semantic information (insulator features) can be correctly recovered with a very high probability, while background areas may experience packet loss or blurring, but this does not affect the machine's classification judgment.

[0071] Image reconstruction: The semantic decoder receives the reconstructed latent vectors and outputs the reconstructed image.

[0072] Final application: The reconstructed image is simultaneously fed into a classification algorithm for fault detection and displayed on the screen for manual review.

[0073] The method described in this embodiment automatically and significantly compresses the background area and reduces background transmission protection when channel bandwidth decreases, thereby squeezing out bandwidth resources to ensure the reconstruction accuracy and transmission reliability of core targets (such as insulators), achieving optimization of human-machine collaborative tasks under dynamic channels. Of course, this invention is not only applicable to UAV power line inspection, but also to human-machine interaction scenarios with extremely high requirements for the reliability of critical information, such as emergency rescue, security monitoring, and telemedicine.

[0074] This invention breaks away from the traditional architecture of separating the source and channel in communication. It extracts pixel-level semantic importance through deep learning and uses this as a priori for source and channel code rate allocation. At the source coding level, it achieves task-aware non-uniform compression, prioritizing the preservation of image regions most critical for machine discrimination and human visual verification under limited resources. At the channel coding level, it constructs a joint resource allocation model based on an improved elite genetic algorithm, introducing innovative "monotonicity constraints" and "gene repair strategies" to ensure stronger channel protection for important data blocks. Ultimately, under dynamically changing wireless channels, the system can adaptively balance machine vision accuracy and image reconstruction quality, significantly improving the reliability and efficiency of operations in human-machine collaborative scenarios.

[0075] This invention proposes a semantically important guided non-uniform coding mechanism. By using a semantic importance mapping map, it forces the system to significantly compress the background region when resources are limited, and concentrates the saved bitrate on the foreground region, which is crucial to downstream tasks. Under conditions of extremely low signal-to-noise ratio or narrow bandwidth, compared with traditional methods, this invention significantly improves the accuracy of machine vision tasks (classification / detection), and can quickly recover key semantic information when only a portion of the bitstream is received at the receiver.

[0076] This invention constructs a source-channel code rate joint allocation model based on latent vector block partitioning. Utilizing an improved elite genetic algorithm, the compression rate and error correction protection level of each block are dynamically adjusted according to the real-time channel state, achieving semantic-level adaptive protection. "Strong protection" is implemented for important blocks, while "weak protection" is applied to less important blocks, ensuring that core services are not interrupted when the channel deteriorates, and significantly reducing the packet error rate of critical information.

[0077] This invention employs a unique three-stage training strategy: rate-distortion, task awareness, and generative adversarial training. In particular, by introducing generative adversarial loss to fine-tune the decoder, it solves the problem that traditional machine-oriented encoding often results in reconstructed images that are visually unappealing to the human eye (severe artifacts). This allows the reconstructed images to be accurately recognized by machines and also used for operator review, perfectly adapting to human-machine collaborative scenarios.

[0078] The above description is merely an embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A semantically important guided image coding and wireless transmission system, characterized in that: It includes a semantic importance analysis module, an adaptive image semantic coding module, a latent vector block module, a source-channel code rate joint allocation module, a code stream grouping module, a channel encoder and a modulator deployed at the transmitting end, and a demodulator, a channel decoder and an adaptive image semantic decoding module deployed at the receiving end. The semantic importance analysis module is used to analyze the importance of each pixel in the input original image X to the downstream machine vision task, and generate a pixel-level semantic importance map. ; The source channel code rate joint allocation module receives the semantic importance mapping. The implicit vector block information and channel state information are used to construct a joint optimization model of source and channel code rate under the total code rate constraint, generating a coding quality vector M and a channel code rate allocation vector r, which respectively guide the semantic encoder to perform task-aware semantic coding and the channel encoder to perform semantic-first channel coding. The adaptive image semantic coding module performs non-uniform, task-related compression coding on the original image based on the coding quality vector M. The adaptive image semantic coding module includes a semantic encoder, a latent vector block module, a quantizer, and an entropy encoder. The semantic encoder employs a deep neural network model trained and converged end-to-end. It takes the original image X and the encoded quality vector M as input, extracts image features adapted to the bit rate and downstream tasks, and outputs a latent vector Z. The latent vector block module divides the latent vector Z into blocks of equal size to obtain the latent vectors. The quantizer processes the hidden vectors after block division. Perform scalar quantization to obtain the latent vector. The entropy encoder processes the quantized latent vectors. Entropy encoding is performed to generate the source code stream L; The code stream grouping module generates code stream groups based on the source code stream L output by the entropy encoder and the channel code rate allocation strategy r, combined with the channel coding group size. And related signaling; this signaling includes channel coding rate and source code stream attribution information, used for channel encoding / decoding and receiver entropy decoding; The channel encoder is responsible for grouping the code stream at the transmitting end according to the channel code rate allocation vector r and signaling. Perform channel coding; The modulator is responsible for modulating the channel-coded packet stream into a baseband signal; The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the demodulator to the receiving end; after receiving the signaling and baseband signal, the receiving end parses the signaling and then performs the following processing: The demodulator demodulates the received baseband signal; The channel decoder performs block decoding based on the parsed signaling to obtain the reconstructed source code stream; The adaptive image semantic decoding module includes a semantic decoder and an entropy decoder. The semantic decoder uses a deep neural network model that has been trained and converged end-to-end. The entropy decoder performs entropy decoding on the code stream output by the channel decoding at the receiving end based on the signaling information to reconstruct the latent vector. The semantic decoder completes image reconstruction based on the reconstructed latent vector.

2. The semantically important guided image coding and wireless transmission system according to claim 1, characterized in that: The source-channel code rate joint allocation module, based on the premise of consistency between the relative positions in the latent vector space and the pixel space, utilizes latent vector block information to map the semantic importance graph. Perform chunking processing, specifically mapping the semantic importance graph. The region is divided into regions aligned with the latent vector block space, and the maximum value within the region is selected as the semantic importance weight of the block. This ensures that pixels and latent vector elements corresponding to the same block share the same semantic weight and coding parameters, thereby outputting the block-based coding quality vector M and the channel code rate allocation vector r.

3. The semantically important guided image coding and wireless transmission system according to claim 1, characterized in that: The adaptive image semantic decoding module and the adaptive image semantic decoding module are trained using a step-by-step optimization strategy: Step T1, Basic Training: Perform pixel-wise rate-distortion optimization iterative training based on multi-rate coding with coding quality vector control. The optimization objective function of the training process is: in, The estimated entropy of the quantized latent vector is given by H and W, where H and W are the height and width of the image, respectively, and HW is the total number of pixels in the image. The weight vector is composed of the mean square error of each pixel. elements in The corresponding elements in the encoded quality vector are processed by a preset monotonically increasing function. Mapped to obtain; Step T2, Task-Aware Training: Based on the convergence of basic training, fix the entropy model parameters, add downstream task-related optimization objective functions, and perform bitrate-distortion-task accuracy optimization training for encoding and decoding, where the task accuracy optimization term is related to the specific downstream task; Step T3, Human Eye Perception Quality Optimization: Freeze the parameters of the semantic encoder and fine-tune only the semantic decoder; introduce a discriminator network and use generative adversarial loss to train the semantic decoder.

4. The semantically important guided image coding and wireless transmission system according to claim 1, characterized in that: In step T2, the added downstream task-related optimization objective function is: , This refers to the loss function corresponding to the downstream machine vision task. The weight of the task loss item.

5. The semantically important guided image coding and wireless transmission system according to claim 1, characterized in that: The source-channel code rate joint allocation module constructs a source-channel code rate joint optimization model under the total code rate constraint based on an elite genetic algorithm, generating a coding quality vector M and a channel code rate allocation vector r, as follows: Step 2-1: The source-channel code rate joint allocation module maps the semantic importance graph. Downsampling is performed to the same dimension as the latent vector blocks, and partitioning is performed based on the block information. The maximum value within each partition is taken as the semantic importance value of that latent vector block, thus obtaining the semantic importance vector. ; Calculate the total bit limit under the current channel conditions using Shannon's formula. ; Step 2-2: Construct the source-channel joint code rate allocation optimization model as follows: ， Among them, the image coding quality vector Divided into N regions, where different elements within a region have the same value. , For discrete variables, their set of values ​​is defined as follows: The latent vector block bitstreams output by the semantic encoder The number of bits is The code length of each packet in the channel coding is... Channel code rate allocation vector , The set of values ​​is defined as ;for the bitstream Channel coding is performed, and the number of effective information bits per packet in the resulting channel code stream is: The error probability for each group is , bitstream The bits after channel coding are The semantic importance of image segmentation lies in the fact that the encoded image blocks must be correctly transmitted to the receiving end to generate revenue. Let $\frac{ ... , where the function Monotone non-decreasing function; Steps 2-3: Use an elite genetic algorithm to solve the source-channel joint code rate allocation optimization model to obtain the image coding quality allocation vector and the channel code rate allocation vector.

6. The semantically important guided image coding and wireless transmission system according to claim 5, characterized in that: Steps 2-3 are as follows: (1) Individual coding: A chromosome consists of two segments: a source code rate allocation segment and a channel code rate allocation segment, and the two segments are independent of each other, meaning they have different gene sets; (2) Definition of fitness function; The fitness function is constructed based on the semantic importance of each block: ， Fit is the expected semantic gain that can be obtained after the image is jointly encoded and transmitted through the source and channel. Based on this fitness function, the optimization objective of the elite genetic algorithm is to maximize Fit while keeping the total transmission bit rate constant. (3) Constraints and optimization of feasible solutions; 1) The total number of bits transmitted does not exceed the channel capacity (Restriction 1): The source channel code rate allocation must also meet the requirement that the total number of bits transmitted through the channel is within the specified range. Less than the channel capacity By adding a penalty to the fitness function, the final fitness function is obtained as follows: ,in, This is a normalization function used to adjust the value of the penalty term. 2) Monotonicity of Source Code Rate and Importance for Each Block (Restriction 2): This allocation mechanism, given a fixed total number of bits transmitted, prioritizes allocating higher source code rates to image blocks with high semantic importance and lower channel code rates to their corresponding bitstreams. This ensures that image blocks with high semantic importance have higher coding accuracy and can be transmitted to the receiver with a higher success rate. Therefore, the source coding rate of each block is a non-decreasing function of the block's semantic importance, and the channel code rate is a non-increasing function of the block's semantic importance. 3) Optimization of feasible solutions: New individuals are generated randomly, and the new individuals must meet the above constraints; After several generations of propagation, the optimal coding quality vector is output. Channel code rate allocation vectors for each hidden vector block .

7. A semantically important guided image coding and wireless transmission method, characterized in that: The method is implemented using a semantic importance-guided image coding and wireless transmission as described in any one of claims 1-6. The method first trains the adaptive image semantic coding module and the adaptive image semantic decoding module using a step-by-step optimization strategy, and then performs the following steps: Step 1: The semantic importance analysis module at the sending end analyzes the importance of each pixel in the input original image X to the downstream machine vision task, and generates a pixel-level semantic importance map. Meanwhile, the latent vector segmentation module generates segmentation information based on the downsampling rate of the semantic encoder and the original image size; Step 2: Map the semantic importance graph The module takes block information and channel state as input, performs source and channel code rate joint allocation, runs the source and channel code rate joint allocation operation, and outputs a coding quality vector for source code rate allocation. Channel code rate allocation vectors for each hidden vector block ,in, The elements in the array take values ​​in the range [0,1] and are divided into N regions; The channel code rate for each block, It is a one-dimensional vector with a length of N; Step 3: Input the original image X and the encoded quality vector M into the semantic encoder to obtain the latent vector Z. Divide the latent vector Z into blocks, and quantize and entropy encode each block to obtain the bitstream of each latent vector block. ; Step 4: The code stream grouping module allocates vectors according to the channel code rate. The code length of the channel coding, for the bit stream The information packets are obtained by grouping them to obtain channel-coded information packets. During the code stream grouping operation, the hidden vector block to which each information group belongs and the channel code rate of the group are recorded. The above recorded information is used as signaling to guide the channel encoder to perform channel coding. Step 5: The channel encoder groups the information bits according to the signaling. Channel coding is performed separately, and the modulator modulates the coded channels into baseband signals. Step 6: The transmitting end sends the signaling generated by the code stream packet module and the baseband signal generated by the modulator to the receiving end; after receiving the signaling and baseband signal, the receiving end performs the following processing: The demodulator at the receiving end demodulates the received signal, the receiving end performs signaling parsing, and the channel decoder performs channel decoding based on the signaling to obtain the image bitstream. Channel decoding: Decoding is performed according to the code rate indicated by the signaling; Image reconstruction: The semantic decoder receives the reconstructed latent vectors and outputs the reconstructed image. .

8. The semantically important guided image coding and wireless transmission method according to claim 7, characterized in that: In step 2, the joint allocation of source and channel code rates is performed based on the elite genetic algorithm, as follows: Step 2-1: The source-channel code rate joint allocation module maps the semantic importance graph. Downsampling is performed to the same dimension as the latent vector blocks, and partitioning is performed based on the block information. The maximum value within each partition is taken as the semantic importance value of that latent vector block, thus obtaining the semantic importance vector. ; Calculate the total bit limit under the current channel conditions using Shannon's formula. ; Step 2-2: Construct the source-channel joint code rate allocation optimization model as follows: ， Among them, the image coding quality vector Divided into N regions, where different elements within a region have the same value. , For discrete variables, their set of values ​​is defined as follows: The number of bits in each latent vector block bitstream output by the semantic encoder is The code length of each packet in the channel coding is... Channel code rate allocation vector , The set of values ​​is defined as ;for the bitstream Channel coding is performed, and the number of effective information bits per packet in the resulting channel code stream is: The error probability for each group is , bitstream After channel coding, the bits are The semantic importance of image segmentation lies in the fact that the encoded image blocks must be correctly transmitted to the receiving end to generate revenue. Let $\frac{ ... , where the function Monotone non-decreasing function; Steps 2-3: Use an elite genetic algorithm to solve the source-channel joint code rate allocation optimization model to obtain the image coding quality allocation vector and the channel code rate allocation vector.

9. The semantically important guided image coding and wireless transmission method according to claim 8, characterized in that: Steps 2-3 are as follows: (1) Individual coding: A chromosome consists of two segments: a source code rate allocation segment and a channel code rate allocation segment, and the two segments are independent of each other, meaning they have different gene sets; (2) Definition of fitness function; The fitness function is constructed based on the semantic importance of each block: ， Fit is the expected semantic gain that can be obtained after the image is jointly encoded and transmitted through the source and channel. Based on this fitness function, the optimization objective of the elite genetic algorithm is to maximize Fit while keeping the total transmission bit rate constant. (3) Constraints and optimization of feasible solutions; 1) The total number of bits transmitted does not exceed the channel capacity (Restriction 1): The source channel code rate allocation must also meet the requirement that the total number of bits transmitted through the channel is within the specified range. Less than the channel capacity By adding a penalty to the fitness function, the final fitness function is obtained as follows: ,in, This is a normalization function used to adjust the value of the penalty term. 2) Monotonicity of Source Code Rate and Importance for Each Block (Restriction 2): This allocation mechanism, given a fixed total number of bits transmitted, prioritizes allocating higher source code rates to image blocks with high semantic importance and lower channel code rates to their corresponding bitstreams. This ensures that image blocks with high semantic importance have higher coding accuracy and can be transmitted to the receiver with a higher success rate. Therefore, the source coding rate of each block is a non-decreasing function of the block's semantic importance, and the channel code rate is a non-increasing function of the block's semantic importance. 3) Optimization of feasible solutions: New individuals are generated randomly, and the new individuals must meet the above constraints; After several generations of propagation, the optimal coding quality vector is output. Channel code rate allocation vectors for each hidden vector block .

10. The semantically important guided image coding and wireless transmission method according to claim 7, characterized in that: The adaptive image semantic decoding module and its training are as follows: Step T1, Basic Training: Perform pixel-wise rate-distortion optimization iterative training based on multi-rate coding controlled by the coding quality vector M. The optimization objective function of the training process is: in, The estimated entropy of the quantized latent vector is given by H and W, where H and W are the height and width of the image, respectively, and HW is the total number of pixels in the image. The weight vector is composed of the mean square error of each pixel. elements in The corresponding elements in the encoded quality vector are processed by a preset monotonically increasing function. Mapped to obtain; Step T2, Task-Aware Training: Based on the convergence of basic training, fix the entropy model parameters, add downstream task-related optimization objective functions, and perform bitrate-distortion-task accuracy optimization training for encoding and decoding, where the task accuracy optimization term is related to the specific downstream task; Step T3, Human Eye Perception Quality Optimization: Freeze the parameters of the semantic encoder and fine-tune only the semantic decoder; introduce a discriminator network and use generative adversarial loss to train the semantic decoder.

Images