An image semantic communication method and system based on semantic segmentation

Abstract

The application provides a kind of image semantic communication method and system based on semantic segmentation, the method comprises: obtaining the image to be transmitted, and determining the application scene type corresponding to the image to be transmitted;The image to be transmitted is carried out semantic segmentation, obtain a plurality of image blocks and the semantic category information and spatial position information corresponding to each image block;Based on the application scene type determined, the semantic category information corresponding to each image block and the channel state information between the sending end and the receiving end, obtain the encoding model parameter corresponding to each image block, so as to determine the encoding model corresponding to each image block based on the encoding model parameter;Each image block is respectively encoded and processed using the corresponding encoding model, and the image block after encoding processing and the spatial position information of each image block are sent to the receiving end, so that the receiving end carries out image reconstruction based on semantic recovery algorithm, completes the semantic communication of the image to be transmitted.The application can realize the reasonable allocation of communication resources.

Description

Technical Field

[0001] This invention relates to the field of semantic communication technology, and in particular to an image semantic communication method and system based on semantic segmentation. Background Technology

[0002] With the development of intelligent sensing, remote interaction, and immersive applications, image data accounts for an increasingly larger proportion of communication systems. In practical applications, communication systems often face problems such as bandwidth limitations, high channel noise, or fluctuating network conditions when transmitting images. How to achieve high-quality image transmission under limited communication resources has become an urgent problem to be solved in the field of communication.

[0003] Traditional image communication methods focus on bit-level fidelity transmission, lacking a deep semantic understanding of image content. In scenarios with limited bandwidth or complex channel conditions, these methods not only transmit a large amount of redundant information, resulting in low communication resource utilization efficiency, but are also highly sensitive to channel noise and bit errors, making it difficult to achieve an effective balance between communication efficiency and image reconstruction quality.

[0004] In recent years, semantic communication technology has gradually attracted attention. Its core idea is to extract semantic information relevant to the receiving end's task at the sending end, transmitting only content valuable to the receiving end's task, thereby significantly reducing communication load and improving system efficiency. Existing semantic communication technologies can be divided into the following two categories: One type is the overall image semantic coding transmission method. This type of method typically employs an end-to-end deep neural network model to extract and encode the overall semantic features of the image to be transmitted, and then performs unified decoding and image reconstruction at the receiving end. However, the overall image semantic coding transmission method usually adopts a uniform semantic feature extraction and encoding strategy for the entire image, which cannot distinguish between core semantic regions that are crucial to the receiving end's task and regions of lower importance. This results in a large amount of limited channel communication resources being consumed by regions that contribute little to the task (such as background regions), while key semantic regions lack sufficient detail during reconstruction, thus affecting both overall communication efficiency and reconstruction quality.

[0005] Another category is semantic communication methods based on semantic segmentation. These methods segment the image to be transmitted into multiple regions with different semantic categories, and then process and transmit each region separately. This type of method has been widely studied and applied due to its advantages in communication accuracy and efficient use of channel resources. However, it still suffers from significant drawbacks, such as the underutilization of semantic region differences, leading to low communication efficiency and reconstruction quality. After segmenting the image into multiple regions with different semantic categories, existing methods often employ coarse-grained semantic coding methods to differentiate between different semantic regions in the entire image (e.g., encoding only regions of interest and non-interest regions), and discard some coding features when channel resources are limited, thus failing to fully utilize image characteristics to match channel resources.

[0006] Therefore, there is an urgent need for an image semantic communication method that can be based on image characteristics to achieve an effective balance between communication efficiency and image reconstruction quality. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide an image semantic communication method and system based on semantic segmentation, which realizes the rational allocation of communication resources, thereby improving the overall communication efficiency and ensuring the reconstruction quality of key semantic regions.

[0008] One aspect of the present invention provides an image semantic communication method based on semantic segmentation, the method comprising the following steps: Acquire the image to be transmitted and determine the application scenario type corresponding to the image to be transmitted; Semantic segmentation is performed on the image to be transmitted to obtain multiple image blocks, semantic category information corresponding to each image block, and spatial location information of each image block; Based on the determined application scenario type, the semantic category information corresponding to each image block, and the channel state information between the sender and receiver, the coding model parameters corresponding to each image block are obtained, and the coding model corresponding to each image block is determined based on the coding model parameters. Each image block is encoded using the corresponding encoding model, and the encoded image blocks and their spatial location information are sent to the receiving end so that the receiving end can reconstruct the image based on the semantic recovery algorithm and complete the semantic communication of the image to be transmitted.

[0009] In some embodiments of the present invention, semantic segmentation of the image to be transmitted includes: Input the image to be transmitted and the determined application scenario type into a pre-trained general semantic segmentation model for semantic segmentation; or Based on the application scenario type, a corresponding dedicated semantic segmentation model is selected from the preset correspondence between application scenario type and dedicated semantic segmentation model, and the image to be transmitted is input into the selected dedicated semantic segmentation model for semantic segmentation. Both the general semantic segmentation model and the specialized semantic segmentation model are trained using semantic category accuracy as the loss function.

[0010] In some embodiments of the present invention, determining the coding model based on coding model parameters for each image block includes: Based on the encoding model parameters, a matching pre-trained encoding model is selected from the preset mapping relationship between encoding model parameters and pre-trained encoding models, and the selected model is used as the corresponding encoding model; or The pre-trained general coding model is adjusted based on the coding model parameters to obtain the corresponding coding model; Both the pre-trained coding model and the pre-trained general coding model are trained using the reconstruction quality of the image samples by the encoder to be trained at the sending end and the decoder to be trained at the receiving end as the loss function.

[0011] In some embodiments of the present invention, when the encoding model is determined based on a preset encoding model parameter-pre-trained encoding model correspondence, the method further includes: sending the identification information of the selected pre-trained encoding model to the receiving end, so that the receiving end can determine the semantic recovery algorithm based on the identification information.

[0012] In some embodiments of the present invention, the coding model parameters of each image block are obtained based on a pre-trained coding decision model; the pre-trained coding decision model is trained using the reconstruction quality of the image samples after semantic segmentation and coding processing by the semantic recovery algorithm at the receiving end as the loss function.

[0013] In some embodiments of the present invention, the channel state information includes one or more of the following: signal-to-noise ratio, channel bandwidth, packet loss rate, and transmission delay. The encoding model parameters include one or more of the following: feature extraction network structure identifier, target dimension of feature vector, compression ratio, and quantization precision.

[0014] In some embodiments of the present invention, the application scenario type corresponding to the image to be transmitted is determined based on a pre-trained scene recognition model; the scene recognition model is constructed based on a classification model.

[0015] Another aspect of the present invention provides an image semantic communication system based on semantic segmentation. The system is deployed at a transmitting end and includes a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the system implements the steps of the method described in any of the above embodiments.

[0016] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0017] Another aspect of the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0018] This invention proposes an image semantic communication method and system based on semantic segmentation. Based on the concept that the importance of the same semantic category varies across different application scenarios, and the required channel resources also differ, this application incorporates the image's application scenario type and channel state information when encoding semantic features. This allows for different encoding strategies for image blocks of different semantic categories. The proposed method can rationally allocate channel resources, thereby achieving an effective balance between communication efficiency and image reconstruction quality.

[0019] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0020] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart illustrating an image semantic communication method based on semantic segmentation in one embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating an image semantic communication method based on semantic segmentation in another embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0024] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0027] Overall image semantic coding and transmission methods typically employ a uniform semantic feature extraction and encoding strategy for the entire image, lacking fine-grained perception of semantic regions within the image, resulting in low communication efficiency and reconstruction quality. While existing semantic communication technologies based on semantic segmentation have improved communication efficiency and image reconstruction quality to some extent, the inventors of this application argue that these technologies overlook the fact that the importance of the same semantic category to changes in communication state can vary significantly across different application scenarios. On one hand, datasets applicable to different application scenarios (such as the ADE20K dataset and the Cityscapes dataset) have different semantic categories and target distributions; on the other hand, the key semantic targets contained in images differ significantly across application scenarios. For example, in autonomous driving scenarios, pedestrians, vehicles, and traffic signs are typically considered key targets, while background areas such as the sky are relatively less important. Similarly, in natural landscape scenarios, natural elements such as mountains, lakes, and vegetation often become the primary focus. Therefore, based on the above analysis, it can be inferred that the importance of the same semantic category varies across different application scenarios.

[0028] To address the aforementioned shortcomings, this invention proposes a novel semantic communication method based on semantic segmentation. This method can be implemented by an image semantic communication system deployed at the sending end. The purpose of this method is to comprehensively consider the image application scenario type, the semantic region category of the image block, and channel state information under the condition of limited communication resources, and to adaptively decide on the semantic feature encoding strategy for different image blocks to achieve reasonable allocation of communication resources, thereby improving overall communication efficiency and ensuring the reconstruction quality of key semantic regions.

[0029] Figure 1This is a flowchart illustrating an image semantic communication method based on semantic segmentation in one embodiment of the present invention. Figure 1 As shown, the method includes steps S110 to S140, as detailed below: Step S110: Obtain the image to be transmitted and determine the application scenario type corresponding to the image to be transmitted. The image to be transmitted can be an image from any application scenario, including but not limited to autonomous driving, medical and health, and geographic information and surveying. Therefore, the image to be transmitted can be a photograph, a painting, or a computer-generated virtual image (such as an animation image).

[0030] In some embodiments of the present invention, the application scenario type corresponding to the image to be transmitted can be determined based on a pre-trained scene recognition model. That is, the image to be transmitted is input into the pre-trained scene recognition model, and the corresponding application scenario type is output. The scene recognition model essentially identifies the application scenario type of an image based on its overall scene attributes. Therefore, a scene recognition model can be constructed based on a classification model, and the constructed model can be trained to obtain a pre-trained scene recognition model. For example, the training process for obtaining the pre-trained scene recognition model can be as follows: A scene recognition dataset (such as ISPRS Vaihingen / Potsdam for remote sensing scenarios, Cityscapes for autonomous driving urban scenarios, and TotalSegmentator for healthcare scenarios) is input as training samples into the scene recognition model to be trained. A loss function is obtained based on the model output and the application scenario type labels in the training samples. The pre-trained scene recognition model is obtained through iteration. The scene recognition dataset needs to include image samples from multiple application scenarios and the corresponding application scenario type labels for the image samples. Furthermore, the application scenario types included in the scene recognition dataset can be designed according to requirements. This application does not specifically limit the type of classification model. The classification model can be a traditional machine learning model such as a vector machine or a random forest, or a deep learning model based on a convolutional neural network structure or an attention mechanism. For example, a classic CNN model (such as ResNet50) pre-trained on a large general image dataset (such as ImageNet) can be used as the feature extraction backbone network, and an adapted classification head (containing a fully connected layer whose output number of neurons precisely corresponds to the number of application scenario types of this method) can be connected after the feature extraction backbone network, so as to adapt it to a specific scenario classification task through fine-tuning.

[0031] Step S120: Perform semantic segmentation on the image to be transmitted to obtain multiple image blocks (the semantic categories of the multiple image blocks obtained in this application are different) and image segmentation information for each image block. The image segmentation information for each image block may include semantic category information, category confidence, spatial location information (which may be segmentation mask information), geometric information, and / or region attribute information, etc. The semantic category information may be the semantic category identifier information of each pixel in the image block; the category confidence indicates the probability that each pixel in the image block belongs to each semantic category; the spatial location information indicates the position of each pixel in the image block in the image to be transmitted; the geometric information may include the boundary contour and / or polygon vertices of the image block; and the region attribute information may include the area of ​​the image block (which can be represented by the number of pixels contained in the image block) and / or the center of the image block (such as centroid coordinates), etc.

[0032] This application can be implemented using a semantic segmentation model in step S120, or it can be performed by combining a classification model and full convolution, or by combining object detection and pixel-level methods. The segmentation methods of the semantic segmentation model will be described in detail below.

[0033] In some embodiments of this invention, existing methods typically employ fixed or single semantic segmentation models (such as U-Net or SegNet) without designing a semantic segmentation scheme based on image characteristics. This may lead to insufficient segmentation accuracy and inaccurate boundaries, thereby affecting the subsequent semantic communication effect. To address the problem of insufficient semantic segmentation accuracy, this application designs a method for accurate semantic segmentation of the image to be transmitted based on the application scenario type.

[0034] More specifically, based on application scenario type segmentation, this application can configure dedicated semantic segmentation models for different application scenario types, or it can adopt a unified general semantic segmentation model. Specifically, the application scenario type determined in step S110 can be used as a label, combined with the image to be transmitted, and input into the pre-trained general semantic segmentation model, so that the general semantic segmentation model can appropriately segment the image to be transmitted for any application scenario type according to the label; or, according to the application scenario type determined in step S110, a corresponding dedicated semantic segmentation model can be selected from the preset correspondence between application scenario type and dedicated semantic segmentation model, and the image to be transmitted can be input into the selected dedicated semantic segmentation model for semantic segmentation. Here, the dedicated semantic segmentation model mentioned in this application refers to a model used for semantic segmentation of the image to be transmitted for a specific application scenario type, while the general semantic segmentation model can input the image to be transmitted for any application scenario type. Furthermore, the mapping between application scenario type and dedicated semantic segmentation model can be based on an index of a pre-configured semantic segmentation model library. The identifier of the dedicated semantic segmentation model is determined according to the application scenario type of the image to be transmitted, and the corresponding dedicated semantic segmentation model is retrieved from the semantic segmentation model library containing multiple pre-trained dedicated semantic segmentation models based on the determined identifier. Dedicated semantic segmentation models matching different application scenario types can be trained using image datasets from the corresponding scenarios, and the network architecture of dedicated semantic segmentation models for different application scenario types can be different. For example, an nnU-Net model trained on the TotalSegmentator dataset can be used as a dedicated semantic segmentation model for medical and health scenarios, while a model formed by training HR-Net+OCR on the ISPRS Vaihingen / Potsdam dataset can be used as a dedicated semantic segmentation model for remote sensing scenarios.

[0035] As an example, the semantic segmentation model mentioned in this application (whether a general semantic segmentation model or a special semantic segmentation model) can be trained using semantic category accuracy as the loss function. That is, the model is iteratively trained by comparing the semantic category labels in the training samples (including image samples and corresponding semantic category labels) with the output of the semantic segmentation model to be trained. Assuming that the semantic segmentation model library contains semantic segmentation models trained on semantic segmentation datasets for different application scenarios, the semantic segmentation dataset must include image samples from different application scenarios, and the semantic category information of each pixel in the image samples as the semantic category label during semantic segmentation model training. This application can also combine semantic segmentation datasets from multiple scenarios used to train the semantic segmentation model to form a dataset for training a scene recognition model. Furthermore, the semantic segmentation model in this application can adopt the same model architecture as existing semantic segmentation models, or it can be custom-designed; this invention is not limited to this.

[0036] This application does not limit the specific type of semantic category. Images from different application scenarios may have different semantic categories, or they may have the same semantic category. Specific categories can be referenced from the classifications in existing semantic segmentation datasets. Furthermore, for multiple image patches with the same semantic category, the subsequent encoding decision model may output the same or similar encoding model parameters.

[0037] Step S130: Based on the determined application scenario type, the semantic category information corresponding to each image patch, and the channel state information (CSI) between the sender and receiver, the coding model parameters (also known as encoder parameters) corresponding to each image patch are obtained. Based on these parameters, the coding model corresponding to each image patch is determined to reflect the semantic value differences of the same semantic category under different scenario conditions and corresponding communication state changes (i.e., enabling image patches of the same semantic category to obtain different coding strategy configurations under different scenarios and communication states). The sender can collect and update the channel state information in real time, or it can acquire the channel state information after acquiring the image to be transmitted; this invention is not limited to these methods. The channel state information may include one or more of signal-to-noise ratio, channel bandwidth, packet loss rate, and transmission delay. The coding model parameters may include one or more of feature extraction network structure identifier, target dimension of feature vector, compression ratio, and quantization accuracy. This application does not limit the specific information types in the channel state information and coding model parameters. The purpose of selecting different coding models for image patches of different semantic categories in this application is to prioritize the feature fidelity and transmission reliability of key semantic regions under limited bandwidth resources, thereby achieving differentiated allocation of communication resources.

[0038] In some embodiments of the present invention, similar to semantic segmentation models, this application may be configured with multiple dedicated coding models (pre-trained) that match specific coding model parameters, or a general pre-trained coding model may be used. Specifically, for each image patch, the process of determining the coding model based on the coding model parameters may be as follows: based on the coding model parameters, a matching pre-trained coding model is selected from the preset correspondence between specific coding model parameters and pre-trained coding models, and the selected model is directly used as the coding model corresponding to the image patch; or the pre-trained general coding model is adjusted based on the coding model parameters to obtain the coding model corresponding to the image patch (no training is required after adjustment). The pre-trained coding model mentioned in this application is a dedicated coding model that matches specific coding model parameters. The preset correspondence between specific coding model parameters and pre-trained coding models may be an index of a pre-configured coding model library (which stores multiple pre-trained dedicated coding models). The specific coding model parameters are matched according to the coding model parameters determined in step S130, the identifier of the matching pre-trained coding model is determined, and the corresponding pre-trained coding model is called from the coding model library according to the determined model identifier. In addition, when the encoding model parameters include multiple fields, the encoding model parameters determined in step S130 may not be able to completely match the specific encoding model parameters in the corresponding relationship. Therefore, in the case where a complete match is not possible, this application designs that the pre-trained encoding model corresponding to the specific encoding model parameter with the most matching field information can be selected as the encoding model determined in step S130, or any pre-trained encoding model corresponding to the specific encoding model parameter can be selected if there is a matching field information.

[0039] As an example, the encoding model mentioned in this application (whether a pre-trained specialized encoding model or a general encoding model) can be trained using the reconstruction quality of image samples by the encoder to be trained at the transmitting end and the decoder to be trained at the receiving end as the loss function. That is, the image sample is input into the encoder to be trained, and the decoder to be trained performs image reconstruction on the encoder's output. The loss function is obtained by comparing the reconstructed image with the image sample input to the encoder, and the trained encoding model (whether a general or specialized model) is iteratively obtained. The encoder to be trained is the pre-trained encoding model, and the semantic recovery algorithm at the receiving end can include a pre-trained decoder (obtained by training the decoder to be trained) and an image patch combination algorithm. The image sample can be an image from a scene recognition dataset or other training images.

[0040] Furthermore, this application can design a pre-trained coding decision model to obtain the coding model parameters for each image patch. Specifically, the determined application scenario type, the semantic category information corresponding to each image patch, and the channel state information between the transmitter and receiver are used as inputs to the pre-trained coding decision model, which can output the coding model parameters corresponding to each image patch. This application does not specifically limit the input data format of the coding decision model. For example, the application scenario type, semantic category information, and channel state information can be converted into feature representations (such as embedding vectors) in the same space through an embedding layer, and these feature representations can be used as input data for the pre-trained coding decision model.

[0041] Existing methods typically assume that the importance of different image regions is determined solely by their semantic category. This leads to the use of the same or coarse-grained semantic feature encoding strategies for semantic regions of different categories under varying channel conditions. This fails to fully utilize the differences between image blocks of different semantic categories for adaptive allocation of communication resources, resulting in degraded reconstruction quality of critical regions or wasted communication resources. The fundamental principle of this application's encoding process is to use the overall scene type of the image and real-time channel state information as prior conditions, semantic segmentation as the structural basis, and image blocks of different semantic categories as the basis for resource allocation. Through an encoding decision model, adaptive communication of image semantic information is achieved.

[0042] As an example, a pre-trained coding decision model can be trained after the scene recognition model, semantic segmentation model, coding model, and semantic recovery algorithm have been trained, in the following way: Obtain a set of image training samples and determine the application scenario type corresponding to each image training sample in the set based on the pre-trained scene recognition model; for each image training sample, use the pre-trained semantic segmentation model (calling a dedicated semantic segmentation model from a semantic segmentation model library or using a general semantic segmentation model) to obtain multiple training sample image patches, the semantic category information corresponding to each training sample image patch, and the spatial location information of each training sample image patch; then input the application scenario type determined by the image training samples, the semantic category information corresponding to each training sample image patch, and the channel state information between the sender and receiver into the coding decision model to be trained to obtain the coding model parameters corresponding to each training sample image patch, thereby enabling the coding decision model to be trained based on each training sample image patch. The encoding model parameters for this image patch determine the encoding model corresponding to each training sample image patch (either by calling a pre-trained dedicated encoding model from the model library or adjusting a pre-trained general encoding model). Each training sample image patch is encoded using the corresponding encoding model, and the encoded training sample image patch and its spatial location information are sent to the receiving end (image patches belonging to the same training image can be packaged and sent together). This allows the receiving end to perform image reconstruction based on the trained semantic recovery algorithm. By comparing each training image in the image training sample set with its corresponding reconstructed image (e.g., calculating the reconstruction error between each image patch's reconstruction result and the original training image, and weighted fusion of the errors of each image patch to construct the overall loss function), the parameters of the encoding decision model can be iteratively updated. This allows the model to gradually learn and generate encoding strategies that improve the reconstruction quality of key regions while satisfying communication resource constraints. In other words, the pre-trained encoding decision model is trained using the reconstruction quality of the semantically segmented and encoded image samples by the receiving end's semantic recovery algorithm as the loss function. In addition, the encoding decision model can be constructed using deep learning models (such as deep reinforcement learning networks or Transformer neural networks built based on algorithms such as deep deterministic policy gradients or deep Q-networks), or parametric function mapping models, etc. This application does not impose specific limitations on the model architecture of the encoding decision model, as long as it can make dynamic decisions on the semantic feature encoding strategy based on the semantic region category, image scene type and communication state information.For example, a coding decision model built on a Transformer neural network may include a feature embedding layer, a self-attention layer, and a policy mapping layer. The feature embedding layer is used to map the input application scenario type, semantic category information of image patches, and channel state information into embedding vectors E1, E2, and E3. The self-attention layer is used to input the fused feature vector after concatenating E1, E2, and E3, and to model the relationship between the three through the self-attention mechanism to generate a feature vector E4 that fuses the information of the three. The policy mapping layer is mainly composed of multilayer perceptrons and is used to output the coding model parameters corresponding to each image patch based on the input feature vector E4.

[0043] The models or algorithms mentioned in this application can be implemented using an end-to-end joint optimization approach or a modular, phased independent optimization approach. The above training methods are merely examples, and this invention is not limited thereto.

[0044] Step S140: As Figure 2 As shown, each image patch can be encoded using a corresponding encoding model (i.e., each image patch is input into the corresponding encoding model), and the encoded image patches (semantic feature-level data, such as encoded feature maps) and the spatial location information of each image patch are sent to the receiving end. This allows the receiving end to reconstruct the image based on the semantic recovery algorithm, completing the semantic communication of the image to be transmitted. The semantic recovery algorithm at the receiving end may include a pre-trained decoder and an image patch combination algorithm. The decoder is used to decode the processed image patches to achieve image patch reconstruction; the image patch combination algorithm is used to assemble the reconstructed image patches according to their spatial location information, thereby generating a complete target image through spatial alignment and fusion. The decoder in the encoding model and semantic recovery algorithm mentioned in this application can be constructed based on generative adversarial networks, diffusion models, or autoregressive models, etc.

[0045] In some embodiments of the present invention, when the encoding model is determined based on a preset mapping between encoding model parameters and pre-trained encoding models, the method proposed in this application further includes: sending the identifier information of the selected pre-trained encoding model to the receiving end, so that the receiving end can determine the decoder in the semantic recovery algorithm based on the identifier information. At this time, the receiving end may be configured with a decoding model library, which stores decoding models corresponding to the encoding models in the encoding model library (the models in the encoding model library and the decoding model library are trained together), and the index of the decoding model library may be a mapping between encoding model identifiers and decoding model identifiers. If a general encoding model is used, the receiving end only needs to store a decoding model that matches the general encoding model (the general encoding model and the decoding model are trained together), and this decoding model does not need to be adjusted according to the encoding model parameters.

[0046] As an example, when image blocks belonging to the same image to be transmitted and their spatial location information are sent to the receiving end at the same time, the processed image block, the spatial location information of the image block, and the identification information of the image to be transmitted to which the image block belongs can be sent to the receiving end together, so that the receiving end can determine the image blocks to be assembled together based on the identification information of the image to be transmitted when reconstructing the image.

[0047] Whether in the semantic segmentation stage or the semantic encoding stage, the method proposed in this application can simultaneously adapt to images of different application scenarios, such as medical images, remote sensing images, and industrial images. Compared with existing technologies, the semantic communication method proposed in this application has the following significant advantages: ① Improved Communication Efficiency: This invention introduces a scene type recognition mechanism at the sending end and adaptively calls a matching dedicated semantic segmentation model based on the application scene type. This makes the division of semantic regions more consistent with the semantic structure features of the corresponding scene, thereby improving the accuracy and stability of semantic segmentation. Precise semantic region division provides a reliable foundation for subsequent region-level semantic feature extraction and encoding, reducing invalid encoding and redundant transmission caused by segmentation errors, thus improving overall communication efficiency.

[0048] ② Improve communication resource utilization efficiency: Based on semantic segmentation, this invention further introduces an encoding decision model, using the semantic category of image blocks, the overall application scenario type of the image, and real-time communication status information as decision-making criteria to dynamically determine the semantic feature encoding strategy parameters for different semantic regions. By explicitly incorporating the region semantic category, overall scenario type, and communication status information into the region-level decision-making process, this invention can employ differentiated feature extraction methods, compression rates, and / or quantization precision for different semantic regions based on current communication conditions. This allows limited communication resources to be preferentially allocated to image regions with higher semantic value based on scenario conditions and communication status, thereby significantly improving communication resource utilization efficiency.

[0049] ③ Ensuring the quality of key region reconstruction: This invention makes separate semantic feature encoding strategies for different semantic regions and allows for different feature extraction structures (different structures result in different feature complexities), feature dimensions, compression rates, and / or quantization precisions for different regions. Therefore, it can employ relatively high-fidelity semantic feature extraction and recovery strategies for key semantic regions even under limited communication resources. Compared to existing methods that use a uniform encoding strategy or coarse-grained encoding for all regions, this invention maintains higher semantic consistency and visual quality in the reconstructed image for key semantic regions through region-level differentiated encoding, while using higher compression rates for secondary regions such as the background. This significantly improves the reconstruction quality of key regions while keeping the overall communication load under control.

[0050] ④ Improve the overall robustness and adaptability of semantic communication: This invention generates and transmits indication information of semantic feature encoding strategies at the sending end, enabling the receiving end to invoke the semantic recovery algorithm corresponding to the encoding strategy based on the indication information, thus achieving coordination between the sending and receiving ends on regional-level semantic processing strategies. This mechanism avoids the receiving end blindly inferring the encoding method, improves decoding stability and system robustness under fluctuating communication states or changing channel conditions, and allows the semantic communication system to maintain relatively stable reconstruction performance in complex communication environments.

[0051] Corresponding to the above method, the present invention also provides an image semantic communication system based on semantic segmentation. The system can be deployed at the transmitting end and includes a computer device. The computer device includes a processor and a memory. The memory stores computer programs / instructions. The processor is used to execute the computer programs / instructions stored in the memory. When the computer programs / instructions are executed by the processor, the system implements the steps of the method described above.

[0052] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0053] This invention also provides a computer program product storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer program product can be a tangible product, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of product known in the art.

[0054] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0055] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0056] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image semantic communication method based on semantic segmentation, characterized in that, The method includes the following steps: Acquire the image to be transmitted and determine the application scenario type corresponding to the image to be transmitted; The image to be transmitted is semantically segmented to obtain multiple image blocks, semantic category information corresponding to each image block, and spatial location information of each image block; Based on the determined application scenario type, the semantic category information corresponding to each image block, and the channel state information between the sending end and the receiving end, the coding model parameters corresponding to each image block are obtained, and the coding model corresponding to each image block is determined based on the coding model parameters. Each image block is encoded using the corresponding encoding model, and the encoded image blocks and their spatial location information are sent to the receiving end so that the receiving end can reconstruct the image based on the semantic recovery algorithm and complete the semantic communication of the image to be transmitted.

2. The method according to claim 1, characterized in that, The semantic segmentation of the image to be transmitted includes: The image to be transmitted and the determined application scenario type are input into a pre-trained general semantic segmentation model for semantic segmentation; or Based on the application scenario type, a corresponding dedicated semantic segmentation model is selected from the preset correspondence between application scenario type and dedicated semantic segmentation model, and the image to be transmitted is input into the selected dedicated semantic segmentation model for semantic segmentation. Both the general semantic segmentation model and the special semantic segmentation model are trained using semantic category accuracy as the loss function.

3. The method according to claim 1, characterized in that, For each image patch, determining the coding model based on the coding model parameters includes: Based on the encoding model parameters, a matching pre-trained encoding model is selected from the preset mapping relationship between encoding model parameters and pre-trained encoding models, and the selected model is used as the corresponding encoding model; or The pre-trained general coding model is adjusted based on the coding model parameters to obtain the corresponding coding model; Both the pre-trained coding model and the pre-trained general coding model are trained using the reconstruction quality of the image samples by the encoder to be trained at the sending end and the decoder to be trained at the receiving end as the loss function.

4. The method according to claim 3, characterized in that, When the encoding model is determined based on the preset encoding model parameter-pre-trained encoding model correspondence, the method further includes: sending the identification information of the selected pre-trained encoding model to the receiving end, so that the receiving end can determine the semantic recovery algorithm based on the identification information.

5. The method according to claim 1, characterized in that, The coding model parameters for each image patch are obtained based on a pre-trained coding decision model. The pre-trained coding decision model is trained using the reconstruction quality of the image samples after semantic segmentation and coding processing by the semantic recovery algorithm at the receiving end as the loss function.

6. The method according to claim 1, characterized in that, The channel state information includes one or more of the following: signal-to-noise ratio, channel bandwidth, packet loss rate, and transmission delay. The encoding model parameters include one or more of the following: feature extraction network structure identifier, target dimension of feature vector, compression ratio, and quantization precision.

7. The method according to claim 1, characterized in that, The application scenario type corresponding to the image to be transmitted is determined based on a pre-trained scene recognition model; the scene recognition model is constructed based on a classification model.

8. An image semantic communication system based on semantic segmentation, the system being deployed at a transmitting end, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images