A method and apparatus for image semantic communication using frequency domain hierarchical finite scalar quantization

Abstract

The application discloses a kind of frequency domain layered finite scalar quantization image semantic communication method and device, belong to communication technical field.The method includes: the target level two-dimensional discrete wavelet transform frequency domain decomposition of original image is carried out, and low-frequency semantic branch and multi-scale high-frequency detail branch are obtained;The low-frequency semantic branch and the multi-scale high-frequency detail branch are respectively subjected to finite scalar quantization processing by semantic communication model, and low-frequency quantization discrete index and multi-scale high-frequency quantization discrete index are obtained;According to channel quality parameter, from the multi-scale high-frequency quantization discrete index, select the target high-frequency quantization discrete index to be transmitted;The low-frequency quantization discrete index and the target high-frequency quantization discrete index are sent to receiving end by wireless channel.This application can improve the robustness of communication transmission, save resource consumption.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to an image semantic communication method and apparatus with frequency domain layered finite scalar quantization. Background Technology

[0002] With the rapid iteration of intelligent wireless communication technology, digital image semantic communication is gradually replacing traditional pixel-level transmission and becoming the core technology for efficient wireless image transmission.

[0003] In existing technologies, digital image semantic communication systems generally adopt learned vector quantization (VQ) schemes to quantize, encode, and index the entire image before transmission. This overall quantization method cannot achieve non-uniform error protection. In scenarios where the channel signal-to-noise ratio decreases and the transmission bandwidth is limited, it is very easy to cause image semantic recognition failure, which cannot meet the core requirements of communication tasks and has low transmission robustness. Moreover, traditional learned VQ relies on a large-scale learnable codebook embedding table, which requires a lot of resources for synchronization and storage at both ends of the receiver. Summary of the Invention

[0004] The purpose of this application is to provide a frequency-domain layered finite scalar quantization image semantic communication method and apparatus, which can solve the problems of low transmission robustness and high resource consumption.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a frequency-domain hierarchical finite scalar quantization method for image semantic communication, which is applied at a transmitting end and includes: The original image is subjected to two-dimensional discrete wavelet transform frequency domain decomposition at the target level to obtain low-frequency semantic branches and multi-scale high-frequency detail branches; The low-frequency semantic branch and the multi-scale high-frequency detail branch are processed by finite scalar quantization using a semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index. Based on the channel quality parameters, the target high-frequency quantization discrete index to be transmitted is selected from the multi-scale high-frequency quantization discrete index. The low-frequency quantization discrete index and the target high-frequency quantization discrete index are transmitted to the receiving end via a wireless channel.

[0006] Secondly, embodiments of this application provide a frequency-domain hierarchical finite scalar quantization image semantic communication method, which is applied at a receiving end and includes: The receiver sends a quantized discrete index, which includes a low-frequency quantized discrete index corresponding to a low-frequency semantic branch and a target high-frequency quantized discrete index corresponding to a target high-frequency detail branch. The low-frequency quantized discrete index and the target high-frequency quantized discrete index are dequantized using a finite scalar model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The low-frequency semantic branch and the target high-frequency detail branch are subjected to inverse wavelet transform processing to obtain the reconstructed image of the original image.

[0007] Thirdly, embodiments of this application provide a frequency-domain hierarchical finite scalar quantization image semantic communication device, applied at a transmitting end, comprising: The frequency domain decoupling module is used to perform target-level two-dimensional discrete wavelet transform frequency domain decomposition on the original image to obtain low-frequency semantic branches and multi-scale high-frequency detail branches. The finite scalar quantization module is used to perform finite scalar quantization processing on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively through the semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index. The transmission scheduling module is used to select the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index according to the channel quality parameters. The transmission module is used to transmit the low-frequency quantization discrete index and the target high-frequency quantization discrete index to the receiving end via a wireless channel.

[0008] Fourthly, embodiments of this application provide a frequency-domain hierarchical finite scalar quantization image semantic communication device, applied at a receiving end, comprising: The receiving module is used to receive the quantized discrete index sent by the sending end. The quantized discrete index includes the low-frequency quantized discrete index corresponding to the low-frequency semantic branch and the target high-frequency quantized discrete index corresponding to the target high-frequency detail branch. The dequantization module is used to perform finite scalar dequantization processing on the low-frequency quantized discrete index and the target high-frequency quantized discrete index through a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The image reconstruction module is used to perform inverse wavelet transform processing on the low-frequency semantic branch and the target high-frequency detail branch to obtain the reconstructed image of the original image.

[0009] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first or second aspect.

[0010] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0011] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or second aspect.

[0012] In this embodiment, by performing a target-level two-dimensional discrete wavelet transform frequency domain decomposition on the original image, a low-frequency semantic branch and a multi-scale high-frequency detail branch are obtained. This physically decouples semantic information from detail information, providing a foundation for differentiated protection and hierarchical transmission. A semantic communication model is used to perform finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch, respectively, to obtain a low-frequency quantized discrete index and a multi-scale high-frequency quantized discrete index. Using finite scalar quantization, the transmitting and receiving ends do not need to synchronously store the codebook embedding table, saving resource consumption. Furthermore, based on channel quality parameters, the target high-frequency quantized discrete index to be transmitted is selected from the multi-scale high-frequency quantized discrete index. The low-frequency quantized discrete index and the target high-frequency quantized discrete index are then transmitted to the receiving end via a wireless channel. This allows for selection of the corresponding semantic branch for transmission based on channel quality. Even in scenarios with poor channel quality, priority can be given to ensuring the normal transmission of the low-frequency quantized discrete index representing the core semantics, improving the robustness of communication transmission. Attached Figure Description

[0013] Figure 1 This is a flowchart of an image semantic communication method using frequency-domain hierarchical finite scalar quantization provided in an embodiment of this application; Figure 2 This is a schematic diagram of the semantic communication model in the embodiments of this application; Figure 3 This is a schematic diagram of codebook synchronization and utilization methods using the learning-based vector quantization method in existing technologies. Figure 4 This is a schematic diagram of the codebook synchronization and utilization method using finite scalar quantization in the embodiments of this application; Figure 5 This is a flowchart of an image semantic communication method using frequency-domain hierarchical finite scalar quantization provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an image semantic communication device with frequency domain hierarchical finite scalar quantization provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an image semantic communication device with frequency domain hierarchical finite scalar quantization provided in an embodiment of this application; Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0016] Learning-based vector quantization schemes have inherent technical drawbacks: First, the overall quantization method deeply couples the core semantic structure of the image with shallow visual detail textures, failing to achieve non-uniform error protection. In scenarios with decreased channel signal-to-noise ratio and limited transmission bandwidth, semantic and detail information degrade simultaneously, easily causing image semantic recognition failure and failing to meet the core requirements of communication tasks. Second, traditional learning-based vector quantization relies on a large-scale learnable codebook embedding table. Both the transmitting and receiving ends need to synchronously store and match complete codebook parameters throughout the process, generating huge synchronization and storage overheads in the megabyte range, making it difficult to adapt to wireless devices with limited storage, computing power, and bandwidth, such as drones, portable terminals, and IoT nodes. Third, learning-based codebooks generally suffer from severe codebook collapse problems, with most code elements remaining inactive and idle for extended periods, resulting in extremely low codebook utilization and significant waste of communication resources. Furthermore, the training process requires additional constraints such as commitment loss, EMA (Exponential Moving Average) updates, and codebook re-initialization, making the training process cumbersome and convergence inefficient. Fourth, existing commercial semantic communication models have a fixed transmission rate. Under different bandwidth and channel conditions, they need to be trained independently, which makes it impossible to achieve multi-rate adaptive transmission of a single model. Their scenario adaptability and engineering practicality are poor, and they are difficult to adapt to complex and ever-changing wireless fading channel environments.

[0017] The frequency-domain hierarchical finite scalar quantization image semantic communication method provided in this application relies on the distribution characteristics of frequency domain information to decouple image features into low-frequency semantic components and high-frequency detail components. It combines branch-differentiated finite scalar quantization (FSQ) with a progressive semantic-first transmission mechanism to achieve low-overhead, highly robust, and adaptive digital image semantic transmission.

[0018] The image semantic communication method and apparatus with frequency domain hierarchical finite scalar quantization provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0019] Figure 1 This is a flowchart of an image semantic communication method using frequency-domain hierarchical finite scalar quantization provided in an embodiment of this application. This frequency-domain hierarchical finite scalar quantization image semantic communication method is applied at the transmitting end, such as... Figure 1 As shown, the frequency domain hierarchical finite scalar quantization image semantic communication method includes steps 110 to 140.

[0020] Step 110: Perform target-level two-dimensional discrete wavelet transform frequency domain decomposition on the original image to obtain low-frequency semantic branches and multi-scale high-frequency detail branches.

[0021] The process involves acquiring the original image to be transmitted, preprocessing it by size normalization and pixel value domain normalization to obtain standardized image data. Then, the standardized image data undergoes a target-level two-dimensional discrete wavelet transform frequency domain decomposition, decoupling it to obtain a single low-frequency semantic branch and a multi-scale high-frequency detail branch. The low-frequency semantic branch carries the core semantic information of the image's global geometric structure and category discrimination, while the high-frequency detail branch carries the image's local texture and edge detail information, achieving physical layering and decoupling of image semantic features and visual detail features. The target level can be set based on requirements, for example, it could be three levels.

[0022] In some embodiments of this application, the two-dimensional discrete wavelet transform includes the Haar wavelet transform; The step of performing a target-level two-dimensional discrete wavelet transform frequency domain decomposition on the original image to obtain a low-frequency semantic branch and a multi-scale high-frequency detail branch includes: performing a target-level Haar wavelet transform frequency domain decomposition on the original image to obtain one approximate sub-band and three directional detail sub-bands corresponding to each level; using the approximate sub-band of the final level as the low-frequency semantic branch; and splicing the three directional detail sub-bands of each level to obtain the high-frequency detail branch at the corresponding scale.

[0023] First, the original image undergoes a first-level Haar wavelet transform frequency domain decomposition, yielding one approximate sub-band and three directional detail sub-bands. Then, the first-level approximate sub-band is subjected to a second-level Haar wavelet transform frequency domain decomposition, yielding another approximate sub-band and three directional detail sub-bands. This process is iteratively repeated, performing Haar wavelet transform frequency domain decomposition on the approximate sub-bands of the previous level according to the corresponding number of levels, until the target decomposition level is reached. The final level's approximate sub-band is used as the low-frequency semantic branch. The three directional detail sub-bands of each level are then concatenated along the channel dimension to obtain the high-frequency detail branches at the corresponding scales. For example, if a 3-level Haar wavelet decomposition is used, three high-frequency detail branches at three scales are ultimately obtained. and 1 low-frequency semantic branch The three-way detail sub-bands include horizontal sub-bands, vertical sub-bands, and diagonal sub-bands.

[0024] The original image is subjected to frequency domain hierarchical decomposition by using the two-dimensional discrete wavelet transform (DWT) of the target level, which separates the image into a single low-frequency semantic branch and multi-scale high-frequency detail branches. The low-frequency branch retains the global geometric structure and core semantics of category discrimination of the image, while the high-frequency branches at each scale carry the texture detail information of the image in the horizontal, vertical and diagonal directions, respectively. This achieves complete decoupling of semantic information and detail information at the physical level, providing a foundation for differentiated protection and hierarchical transmission.

[0025] Step 120: Perform finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively through the semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index.

[0026] The semantic communication model can include a model part for the sending end and a model part for the receiving end.

[0027] By performing finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch in the model used at the sending end of the semantic communication model, low-frequency quantized discrete indexes and multi-scale high-frequency quantized discrete indexes are obtained. The quantization level used when quantizing the low-frequency semantic branch can be the same as or different from the quantization level used when quantizing the high-frequency semantic branch. For example, the quantization level used when quantizing the low-frequency semantic branch can be higher than the quantization level used when quantizing the high-frequency semantic branch.

[0028] Step 130: Select the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index according to the channel quality parameters.

[0029] The system collects the channel quality parameters of the current wireless channel in real time. Based on these parameters, it dynamically selects the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index. The target high-frequency quantization discrete index can be at least one of the high-frequency quantization discrete indices in the multi-scale high-frequency quantization discrete index, or it can be empty, meaning no high-frequency quantization discrete index will be transmitted.

[0030] Channel quality parameters characterize the transmission quality of the current wireless channel. If the transmission quality of the current wireless channel is determined to be good based on the channel quality parameters, all high-frequency quantization discrete indices can be transmitted. If the transmission quality of the current wireless channel is determined to be poor based on the channel quality parameters, high-frequency quantization discrete indices do not need to be transmitted. If the transmission quality of the current wireless channel is determined to be between poor and good based on the channel quality parameters, one or more high-frequency quantization discrete indices can be selected from the multi-scale high-frequency quantization discrete indices for transmission.

[0031] In some embodiments of this application, the channel quality parameters include the signal-to-noise ratio; The step of selecting the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index according to the channel quality parameters includes: comparing the signal-to-noise ratio (SNR) with a plurality of gradient-increasing SNR thresholds respectively, determining the target number of SNR thresholds smaller than the SNR, wherein the total number of gradient-increasing SNR thresholds is the same as the target level; and selecting the target number of high-frequency quantization discrete indices from the multi-scale high-frequency quantization discrete index in descending order of transmission priority as the target high-frequency quantization discrete index.

[0032] Taking a level 3 wavelet transform as an example, the decomposition yields three high-frequency detail branches, and incremental signal-to-noise ratio thresholds are calibrated on the validation set. The signal-to-noise ratio of the real-time wireless channel is The target number of high-frequency branches transmitted is:

[0033] in, Indicates the number of steps in ascending order of gradient. l A signal-to-noise ratio threshold.

[0034] The above formula means that the number of targets is the number of signal-to-noise ratio thresholds that are smaller than the signal-to-noise ratio of the wireless channel.

[0035] With the signal-to-noise ratio threshold of the AWGN channel For example, dB: when At dB, only the low-frequency semantic branch is transmitted. The corresponding low-frequency quantized discrete index; when At dB, transmit low-frequency semantic branch The corresponding low-frequency quantization discrete index and the third-level high-frequency detail branch The corresponding high-frequency quantized discrete index; when At dB, transmit low-frequency semantic branch The corresponding low-frequency quantization discrete index and the third-level high-frequency detail branch The corresponding high-frequency quantization discrete index and the second-level high-frequency detail branch The corresponding high-frequency quantized discrete index; when At dB, transmit low-frequency semantic branch The corresponding low-frequency quantization discrete index and the third-level high-frequency detail branch The corresponding high-frequency quantization discrete index, and the second-level high-frequency detail branch. The corresponding high-frequency quantized discrete index, and the first-level high-frequency detail branch. The corresponding high-frequency quantization discrete index. This example uses a coarse-to-fine detail branch activation order; in a real system, the branch energy, classification gain, and reconstruction gain on the validation set can also be used to adjust the index. The transmission priority is calibrated, and then the target high-frequency quantized discrete index to be transmitted is selected according to the transmission priority during transmission.

[0036] By presetting multiple sets of gradient-increasing signal-to-noise ratio (SNR) thresholds, the real-time detected channel SNR is compared with each threshold to select the target high-frequency quantized discrete index to be transmitted. The higher the channel SNR, the more high-frequency detail branches are selected, and the higher the image reconstruction accuracy. The lower the channel SNR, the more high-frequency detail branches are removed step by step to reduce the transmission bit rate and achieve multi-rate adaptive transmission of a single model without retraining.

[0037] Step 140: The low-frequency quantization discrete index and the target high-frequency quantization discrete index are transmitted to the receiving end via a wireless channel.

[0038] The low-frequency semantic branch is set as a fixed, mandatory branch, and combined with the filtered high-frequency detail branches to form the optimal transmission branch set corresponding to the current channel. The low-frequency quantization discrete index and the target high-frequency quantization discrete index corresponding to the optimal transmission branch set are transmitted to the receiver via a wireless channel. The wireless channel may include an AWGN (Additive White Gaussian Noise) channel and / or a Rayleigh fading channel.

[0039] The image semantic communication method with frequency-domain hierarchical finite scalar quantization provided in this application decomposes the original image into a target-level two-dimensional discrete wavelet transform in the frequency domain, obtaining a low-frequency semantic branch and a multi-scale high-frequency detail branch. This physically decouples semantic information from detail information, providing a foundation for differentiated protection and hierarchical transmission. The semantic communication model performs finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively, obtaining a low-frequency quantized discrete index and a multi-scale high-frequency quantized discrete index. Using finite scalar quantization, the transmitting and receiving ends do not need to synchronously store codebook embedding tables, saving resources. Furthermore, based on channel quality parameters, the target high-frequency quantized discrete index to be transmitted is selected from the multi-scale high-frequency quantized discrete index. The low-frequency quantized discrete index and the target high-frequency discrete index are then transmitted to the receiving end via a wireless channel. This allows for selection of the corresponding semantic branch for transmission based on channel quality. Even in scenarios with poor channel quality, priority is given to ensuring the normal transmission of the low-frequency quantized discrete index representing the core semantics, improving the robustness of communication transmission.

[0040] In some embodiments of this application, the semantic communication model includes: a first branch encoder and a first finite scalar quantization module corresponding to the low-frequency semantic branch, and a second branch encoder and a second finite scalar quantization module corresponding to each scale high-frequency detail branch respectively; The step of performing finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index includes: The low-frequency semantic branch is encoded by the first branch encoder to obtain low-frequency hidden layer features; For each high-frequency detail branch at each scale, the high-frequency detail branch is encoded by the second branch encoder corresponding to the scale to obtain high-frequency hidden layer features; The first finite scalar quantization module performs finite scalar quantization on the low-frequency hidden layer features according to the first quantization level corresponding to the low-frequency semantic branch to obtain the low-frequency quantized discrete index. For each high-frequency detail branch at each scale, the high-frequency hidden layer features at the scale are processed by the second finite scalar quantization module corresponding to the scale according to the second quantization level corresponding to the high-frequency detail branch, so as to obtain the high-frequency quantized discrete index of the scale. The first quantization level is higher than the second quantization level.

[0041] Figure 2 This is a schematic diagram of the semantic communication model in the embodiments of this application, such as... Figure 2 As shown, the semantic communication model may include an encoder, a finite scalar quantization module (FSQ), and a finite scalar dequantization module (FSQ). Figure 2 The encoder includes an inverse quantization module, a decoder, and a classifier. The encoder may include a first branch encoder corresponding to the low-frequency semantic branch and a second branch encoder corresponding to the high-frequency detail branches at each scale. The finite scalar quantization module may include a first finite scalar quantization module corresponding to the low-frequency semantic branch and a second finite scalar quantization module corresponding to the high-frequency detail branches at each scale.

[0042] A differentiated finite scalar quantization strategy is adopted for the low-frequency semantic branch and the high-frequency detail branches at each scale to construct a storageless implicit quantization codebook, and to complete the discrete quantization processing of the features of each branch, and output the quantization discrete index corresponding to each branch; the low-frequency semantic branch is configured with a high-precision quantization level, and the high-frequency detail branches are configured with a low-precision quantization level to reduce the quantization distortion of semantic features.

[0043] By configuring asymmetric quantization levels, the quantization interval of semantic branches can be reduced, semantic quantization distortion can be decreased, and the ability of semantic branches to resist channel disturbances can be improved.

[0044] like Figure 2 As shown, after frequency domain decomposition of the original image using Discrete Wavelet Transform (DWT), the low-frequency semantic branch A is processed by the first branch encoder. J Encoding is performed to obtain low-frequency hidden layer features. These features are then input into the first finite scalar quantization module (FSQ) corresponding to the low-frequency semantic branch. Based on the first quantization level, the FSQ module performs finite scalar quantization on the low-frequency hidden layer features to obtain the low-frequency quantized discrete index i. s For each scale, the high-frequency detail branch ( Figure 2 Only B1 and B2 are shown in the image. The high-frequency detail branches are encoded by the second branch encoder (E1, E2) corresponding to this scale to obtain high-frequency hidden layer features. These features are then input into the second finite scalar quantization module at the corresponding scale. The second finite scalar quantization module performs finite scalar quantization processing on the high-frequency hidden layer features based on the second quantization level to obtain the high-frequency quantized discrete index. i 1. i 2.

[0045] The low-frequency semantic branch carries category discrimination and global structural information, therefore it uses a larger quantization level vector, such as the first quantization level vector. Its implicit codebook capacity is The high-frequency detail branch mainly carries texture and local edges, and can use a smaller quantization level vector, such as the second quantization level vector. Its implicit codebook capacity is Within the same value range, the higher the quantization level, the smaller the quantization interval, and the lower the quantization error.

[0046] By configuring an asymmetric quantization level where the first quantization level is higher than the second quantization level, the quantization interval of the semantic branch can be reduced, semantic quantization distortion can be decreased, and the semantic branch's resistance to channel disturbances can be improved.

[0047] In some embodiments of this application, the step of performing finite scalar quantization on the low-frequency hidden layer features according to the first quantization level corresponding to the low-frequency semantic branch by the first finite scalar quantization module to obtain the low-frequency quantized discrete index includes: determining a first quantization interval according to the first quantization level by the first finite scalar quantization module, and performing a rounding operation on the low-frequency hidden layer features according to the first quantization interval and the first feature channel value range constrained by the network parameters of the first finite scalar quantization module to obtain the low-frequency quantized discrete index.

[0048] The first finite scalar quantization module constrains the value range of the first feature channel by combining a learnable scaling parameter and a learnable slope parameter with a hyperbolic tangent activation function. It calculates a uniform first quantization interval based on the first quantization level, and then performs a rounding operation on the low-frequency hidden layer features based on the first quantization interval and the value range of the first feature channel, achieving discrete quantization of the features and obtaining a low-frequency quantized discrete index. During training, the first finite scalar quantization module can use a pass-through estimator to complete the model's backpropagation gradient transfer, eliminating the need to construct a learnable embedding codebook table, configure a commitment loss function, or perform EMA codebook updates and codebook reinitialization operations.

[0049] In some embodiments of this application, the step of performing finite scalar quantization on the high-frequency hidden layer features of the scale according to the second quantization level corresponding to the high-frequency detail branch for each scale to obtain the high-frequency quantized discrete index of the scale includes: for each high-frequency detail branch, determining a second quantization interval according to the second quantization level through the second finite scalar quantization module corresponding to the scale, and performing a rounding operation on the high-frequency hidden layer features of the scale according to the second quantization interval and the second feature channel value range constrained by the network parameters of the second finite scalar quantization module to obtain the high-frequency quantized discrete index of the scale.

[0050] The second finite scalar quantization module constrains the range of values ​​for the second feature channel by combining a learnable scaling parameter and a learnable slope parameter with a hyperbolic tangent activation function. For each scale's high-frequency detail branch, the second finite scalar quantization module calculates a uniform second quantization interval based on the second quantization level. Then, based on the second quantization interval and the range of values ​​for the second feature channel, it performs a rounding operation on the high-frequency hidden layer features, achieving discrete quantization of the features and obtaining a high-frequency quantized discrete index. During training, the second finite scalar quantization module can use a pass-through estimator to complete the model's backpropagation gradient transfer, eliminating the need to construct a learnable embedding codebook table, configure a commitment loss function, or perform EMA codebook updates and codebook reinitialization operations.

[0051] Figure 3 This is a schematic diagram illustrating the codebook synchronization and utilization method using the learned vector quantization approach in existing technologies. Figure 4 This is a schematic diagram illustrating the codebook synchronization and utilization method using finite scalar quantization in an embodiment of this application. For example... Figure 3 As shown, learned vector quantization requires explicit codebook embedding. Both the sending and receiving ends need to synchronize complete codebook parameters, and the storage space occupied by the sending and receiving ends... It is 512KB, and there may be a large number of inactive codewords after training; for example Figure 4 As shown, this application employs branched finite scalar quantization, constraining each channel to a finite scalar level and forming an implicit codebook through Cartesian product. The transmitting and receiving ends only need to synchronize the level vector, scaling parameter, and slope parameter of each branch, thus minimizing storage space usage. With a size of 0.03KB, this method eliminates the processes of nearest neighbor codebook lookup, commitment loss, EMA codebook update, and codebook reinitialization, making it suitable for resource-constrained digital semantic communication terminal deployments.

[0052] This embodiment of the application employs implicit finite scalar quantization instead of learned codebook quantization, eliminating the need to store large-scale codebook embedding tables. The synchronization metadata overhead at both ends is reduced from the traditional 500KB level to 0.03KB level, with synchronization overhead nearly zero. This completely solves the problems of high storage overhead and high synchronization costs in traditional solutions, perfectly adapting to the deployment of various resource-constrained lightweight wireless terminal devices. Furthermore, in this embodiment, all quantization grid nodes are valid symbols, achieving 100% codebook activation utilization, completely eliminating the codebook collapse, symbol idleness, and resource waste problems commonly found in traditional VQ.

[0053] In some embodiments of this application, the semantic communication model further includes: a finite scalar dequantization module and a decoder located at the receiving end; The training process of the semantic communication model includes: In the absence of a channel, the initial semantic communication model is pre-trained based on training samples to obtain a pre-trained semantic communication model. With the addition of a wireless channel, the pre-trained semantic communication model is trained based on the training samples, and all high-frequency detail branches are transmitted during the training process to obtain an intermediate semantic communication model. With the addition of a wireless channel, the intermediate semantic communication model is trained based on the training samples, and during the training process, the high-frequency detail branch to be transmitted is selected according to the signal-to-noise ratio of the wireless channel to obtain the trained semantic communication model.

[0054] The finite scalar dequantization module at the receiving end performs dequantization processing on the received quantized discrete index, and the decoder decodes the low-frequency hidden layer features and target high-frequency hidden layer features obtained after dequantization by the finite scalar dequantization module. The semantic communication model also includes a classifier at the receiving end, which performs semantic classification on the low-frequency semantic features.

[0055] The semantic communication model can be trained using a three-stage training mechanism. The first stage disables channel noise. In the absence of a channel, the branch encoder, finite scalar quantization module, decoder, and finite scalar dequantization module of the initial semantic communication model are pre-trained based on training samples. Specifically, the branch encoder and the priority scalar quantization module process the training samples sequentially to obtain low-frequency quantized discrete indices and high-frequency quantized discrete indices for multi-scale samples. Then, the finite scalar dequantization module and the decoder process these indices to obtain the reconstructed image. During this process, backpropagation based on a composite loss function is used to update the network parameters. Through this first stage of training, stable separation of each frequency domain branch can be achieved.

[0056] The second stage introduces digital modulation and channel perturbation. Specifically, with the addition of a wireless channel, the pre-trained semantic communication model's branch encoder, finite scalar quantization module, decoder, and finite scalar dequantization module are trained based on training samples. The branch encoder and priority scalar quantization module sequentially process the training samples to obtain low-frequency quantized discrete indices and multi-scale high-frequency quantized discrete indices. These indices are then subjected to channel modulation and perturbation, followed by demodulation to obtain the received low-frequency and high-frequency quantized discrete indices. Finally, the finite scalar dequantization module and decoder process these indices to obtain the reconstructed image. During this process, backpropagation based on a composite loss function updates the network parameters. This second stage of training regularizes the distribution of decomposition features in each branch.

[0057] The third stage involves random branch dropout fine-tuning under multi-signal-to-noise ratio (SNR) conditions. Specifically, with the wireless channel added, the branch encoder, finite scalar quantization module, decoder, and finite scalar dequantization module in the intermediate semantic communication model are trained based on training samples. The branch encoder and priority scalar quantization module process the training samples sequentially to obtain low-frequency quantized discrete indices and multi-scale high-frequency quantized discrete indices. The target sample's high-frequency quantized discrete index is selected based on the wireless channel's SNR. Channel modulation and perturbation are applied to both the low-frequency and high-frequency quantized discrete indices, followed by demodulation to obtain the received low-frequency and high-frequency quantized discrete indices. These are then processed by the finite scalar dequantization module and decoder to obtain the reconstructed image. Backpropagation based on a composite loss function updates the network parameters during this process. This third stage of training enhances the semantic communication model's dynamic rate adaptation capability and channel robustness.

[0058] During the training process in the above stages, when performing backpropagation, the pass-through estimator is used to complete the backpropagation gradient transfer of the model for the finite scalar quantization module. There is no need to build a learnable embedding codebook table, configure a commitment loss function, or perform EMA codebook update and codebook reinitialization operations.

[0059] Based on the third-stage training, the embodiments of this application can achieve single-model retraining and multi-rate adaptive transmission. By dynamically adding or removing high-frequency detail branches, it can adapt to multiple transmission bit rates. There is no need to train the model independently for different bandwidths and channel conditions, which greatly reduces the model iteration cost and engineering deployment cost. It effectively solves the problems of poor scenario adaptability and weak scalability of traditional models and can adapt to dynamically changing wireless channel environments.

[0060] In some embodiments of this application, the semantic communication model employs a composite loss function during training, which includes a pixel reconstruction loss function, a visual perception loss function, a semantic classification cross-entropy loss function, and a frequency domain consistency loss function.

[0061] The composite loss function can be composed of a weighted combination of a pixel reconstruction loss function, a visual perception loss function, a semantic classification cross-entropy loss function, and a frequency domain consistency loss function. Specifically, the pixel reconstruction loss function constrains the accuracy of image pixel reconstruction; the visual perception loss function optimizes the visual effect for the human eye; the semantic classification cross-entropy loss function supervises the multi-class semantic classification ability of low-frequency semantic branches to ensure semantic recognition accuracy; and the frequency domain consistency loss function constrains the restoration accuracy of features in each frequency domain branch.

[0062] The composite loss function can be expressed as follows:

[0063] in, This represents the pixel reconstruction loss function. , x This represents the original image in the training samples. Indicates the reconstructed image. The weights represent the pixel reconstruction loss function. Represents the visual perception loss function. , This represents a pre-trained visual feature extraction network. The weights represent the visual perception loss function. This represents the semantic classification cross-entropy loss function. , This represents the indicator of the true class label of the training sample in class c (i.e., the one-hot label component). This represents the predicted probability of the receiver classifier for the c-th class output. The weights represent the weights of the semantic classification cross-entropy loss function; Represents the frequency domain consistency loss function. The weights represent the frequency domain consistency loss function.

[0064] The frequency domain consistency loss function is expressed as follows:

[0065] in, This represents the j-th level high-frequency detail branch. This represents the j-th high-frequency detail branch obtained from the reconstruction, where J represents the target level. This represents the three-way subband of the j-th level obtained from the reconstruction.

[0066] This application simplifies the model training process by removing redundant mechanisms such as commitment loss, EMA codebook updates, and periodic codebook reinitialization required by traditional VQ, reducing training computational consumption and debugging difficulty, and improving model training stability and convergence speed. Simultaneously, it employs a multi-weighted composite loss function and a three-stage course training strategy, balancing image pixel reconstruction accuracy, visual perception effect, semantic classification accuracy, and frequency domain feature consistency, resulting in superior overall model performance. This application simplifies semantic branching through high-precision asymmetric quantization configuration, strictly controlling semantic quantization distortion errors. Combined with semantic classification invariance constraints, it reserves sufficient channel disturbance error budget, effectively suppressing semantic drift caused by noise and fading, ensuring the accuracy and stability of image semantic recognition under various channel conditions. It is widely adaptable to mainstream wireless channels such as AWGN and Rayleigh fading, exhibiting high versatility, practicality, and industrialization value.

[0067] Figure 5 This is a flowchart of an image semantic communication method using frequency-domain hierarchical finite scalar quantization provided in an embodiment of this application. This frequency-domain hierarchical finite scalar quantization image semantic communication method is applied at the receiving end, such as... Figure 5 As shown, the frequency domain hierarchical finite scalar quantization image semantic communication method includes steps 510 to 530.

[0068] Step 510: Receive the quantized discrete index sent by the sending end. The quantized discrete index includes the low-frequency quantized discrete index corresponding to the low-frequency semantic branch and the target high-frequency quantized discrete index corresponding to the target high-frequency detail branch.

[0069] The target high-frequency detail branch is one or more high-frequency detail branches selected by the transmitter from all high-frequency detail branches based on the channel quality parameters. In the case of poor channel quality parameters, the target high-frequency detail branch may not include any high-frequency detail branches, that is, the target high-frequency detail branch is empty, which means that the quantization discrete index received by the receiver only includes the low-frequency quantization discrete index.

[0070] Step 520: Dequantize the low-frequency quantized discrete index and the target high-frequency quantized discrete index using a semantic communication model with finite scalars to obtain the low-frequency semantic branch and the target high-frequency detail branch.

[0071] The semantic communication model can include a model part for the sending end and a model part for the receiving end.

[0072] The low-frequency quantized discrete index and the target high-frequency quantized discrete index are dequantized using finite scalars in the model part of the semantic communication model for the receiver, respectively, to obtain the low-frequency semantic branch and the target high-frequency detail branch. The quantization level used when dequantizing the low-frequency quantized discrete index can be the same as or different from the quantization level used when dequantizing the high-frequency quantized discrete index. For example, the quantization level used when dequantizing the low-frequency quantized discrete index can be higher than the quantization level used when dequantizing the high-frequency quantized discrete index.

[0073] Step 530: Perform inverse wavelet transform processing on the low-frequency semantic branch and the target high-frequency detail branch to obtain the reconstructed image of the original image.

[0074] The low-frequency semantic branch and the target high-frequency detail branch are processed by inverse wavelet transform, and the branches are fused to obtain the reconstructed image of the original image. For the high-frequency detail branch that has not been transmitted, zero padding or empty branches agreed upon during the training phase can be used as placeholders to achieve progressive reconstruction.

[0075] The frequency-domain hierarchical finite scalar quantization image semantic communication method provided in this application receives a quantized discrete index sent by the transmitting end. The quantized discrete index includes a low-frequency quantized discrete index corresponding to a low-frequency semantic branch and a target high-frequency quantized discrete index corresponding to a target high-frequency detail branch. The low-frequency quantized discrete index and the target high-frequency quantized discrete index are dequantized using a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The low-frequency semantic branch and the target high-frequency detail branch are then subjected to inverse wavelet transform processing to obtain the reconstructed image of the original image. By using finite scalar quantization, the transmitting and receiving ends do not need to synchronously store the codebook embedding table, which can save resource consumption. Moreover, the target high-frequency quantized discrete index is a partial high-frequency quantized discrete index selected by the transmitting end from the multi-scale high-frequency detail branches corresponding to the multi-scale quantized discrete indexes obtained by wavelet transform decomposition based on channel quality parameters. In this way, the corresponding semantic branch can be selected for transmission based on channel quality. In scenarios with poor channel quality, the low-frequency quantized discrete index representing the core semantics can be transmitted normally first, which can improve the robustness of communication transmission.

[0076] In some embodiments of this application, the semantic communication model includes a finite scalar dequantization module and a reconstruction decoder. The finite scalar dequantization module includes a first finite scalar dequantization module corresponding to the low-frequency semantic branch and a second finite scalar dequantization module corresponding to the high-frequency detail branches at each scale. The step of performing finite scalar dequantization processing on the low-frequency quantized discrete index and the target high-frequency quantized discrete index using a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch includes: performing finite scalar dequantization processing on the low-frequency quantized discrete index using a first finite scalar dequantization module to obtain the restored low-frequency hidden layer features; performing finite scalar dequantization processing on the target high-frequency quantized discrete index using a second finite scalar dequantization module corresponding to the target high-frequency detail branch to obtain the restored target high-frequency hidden layer features; and decoding the restored low-frequency hidden layer features and the restored target high-frequency hidden layer features using the reconstruction decoder to obtain the low-frequency semantic branch and the target high-frequency detail branch.

[0077] The model portion for the receiver in the semantic communication model may include a finite scalar dequantization module and a reconstruction decoder. The finite scalar dequantization module includes dequantization modules corresponding to each branch, specifically a first finite scalar dequantization module corresponding to the low-frequency semantic branch and a second finite scalar dequantization module corresponding to the high-frequency detail branches at each scale. The reconstruction decoder may include branch decoders or joint reconstruction decoders corresponding to each branch, and the joint reconstruction decoder can jointly decode the low-frequency semantic branch and the target high-frequency semantic branch.

[0078] like Figure 2 As shown, the low-frequency quantized discrete index is dequantized using a first finite scalar dequantization module based on a first quantization level to obtain the restored low-frequency hidden layer features; the low-frequency hidden layer features are then obtained using a second finite scalar dequantization module corresponding to the high-frequency detail branch. Figure 2 (The diagram shows inverse quantization) The corresponding target high-frequency quantized discretization is dequantized using finite scalars to obtain the restored target high-frequency hidden features. The restored low-frequency hidden features and the restored target high-frequency hidden features are then decoded by a reconstruction decoder to obtain the low-frequency semantic branch and the target high-frequency detail branch. For example, the restored low-frequency hidden features can be decoded using the branch decoder corresponding to the low-frequency semantic branch to obtain the restored low-frequency semantic branch, and the restored target high-frequency hidden features can be decoded using the branch decoder corresponding to the target high-frequency semantic branch to obtain the restored target high-frequency semantic branch; alternatively, the restored low-frequency hidden features and the restored target high-frequency hidden features can be jointly decoded using a joint reconstruction decoder to obtain the restored low-frequency semantic branch and the restored target high-frequency detail branch.

[0079] By employing implicit finite scalar quantization instead of learned codebook quantization, the need for storing large-scale codebook embedding tables is eliminated. The synchronization metadata overhead at both ends is reduced from the traditional 500KB level to 0.03KB level, with synchronization overhead nearly zero. This effectively solves the problems of high storage overhead and high synchronization costs associated with traditional solutions, perfectly adapting to the deployment of various resource-constrained lightweight wireless terminal devices. Furthermore, in this embodiment, all quantization grid nodes are valid symbols, achieving 100% codebook activation utilization, thus eliminating the codebook collapse, symbol idleness, and resource waste problems commonly found in traditional VQ.

[0080] In some embodiments of this application, the semantic communication model further includes a semantic decoder and a classifier; The method further includes: decoding the low-frequency hidden layer features using the semantic decoder to obtain semantic features; and performing semantic classification on the semantic features using the classifier to obtain a classification result.

[0081] like Figure 2 As shown, the semantic communication model also includes a semantic decoder A. J and classifier g (.).

[0082] Based on the principle of semantic classification invariance, the sum of quantization distortion error and channel disturbance error in the low-frequency semantic branch is verified. When the total error is less than a preset semantic error threshold, the reconstructed image semantics is deemed valid, and digital image semantic communication transmission is completed. For example, during training, the classification result can be substituted into the cross-entropy loss function, and the result can be used as the total error. In application, the maximum confidence level in each category can be determined from the classification results, and the difference between the integer 1 and the maximum confidence level can be used as the total error.

[0083] Semantic validity determination, as a semantic reliability verification step at the receiver, can be performed after each reconstructed output or triggered according to the service reliability level. The determination primarily targets the total disturbance received by the low-frequency semantic branch, including quantization distortion error and channel disturbance error. When the total error is less than a preset semantic error threshold, the classifier output is marked as semantically valid; when the total disturbance error is greater than or equal to the semantic error threshold, the classifier output is marked as semantically invalid. If semantically invalid, the receiver can return a retransmission request or incremental transmission request, and the scheduler can increase the protection level of the low-frequency semantic branch, retransmit more high-frequency detail branches, or resend key indices. In scenarios without feedback links, the receiver outputs a low-confidence label and uses the classification result of the low-frequency semantic branch.

[0084] This application also provides a semantic communication system, which may include an image preprocessing module, a frequency domain decoupling module, a branch finite scalar quantization module, a dynamic transmission scheduling module, a wireless channel transmission module, an image reconstruction module, and a semantic fidelity verification module. The image preprocessing module is used to perform size normalization and pixel normalization preprocessing on the original image. The frequency domain decoupling module is used to achieve frequency domain hierarchical decoupling between the image semantic branches and multi-scale detail branches through multi-level discrete wavelet transform. The branch finite scalar quantization module is used to perform differentiated implicit grid quantization on different frequency domain branches to generate discrete quantization indexes. The dynamic transmission scheduling module is used to dynamically adjust the number of high-frequency detail branches transmitted according to real-time channel quality to achieve semantic-priority progressive transmission. The wireless channel transmission module is used to complete the digital channel transmission of the quantization index. The image reconstruction module is used to realize branch dequantization and inverse wavelet transform image reconstruction. The semantic fidelity verification module is used to verify the total error of the semantic branches to ensure the stability and effectiveness of the image semantic classification results.

[0085] The semantic communication method provided in this application includes image preprocessing, multi-level wavelet frequency domain decoupling, branch-differentiated finite scalar quantization, channel quality-aware progressive scheduling, digital index transmission, receiver reconstruction, and semantic fidelity verification steps. First, the transmitter performs size normalization preprocessing on the input image to obtain standardized image data. Then, the image is decomposed in the frequency domain using a preset level of two-dimensional discrete wavelet transform (DWT) to separate a single low-frequency semantic branch and multi-scale high-frequency detail branches. The low-frequency semantic branch retains the global geometric structure and core semantics of category discrimination, while each scale high-frequency detail branch carries texture detail information in the horizontal, vertical, and diagonal directions of the image, respectively. This achieves complete decoupling of semantic information and detail information at the physical level, providing a foundation for differentiated protection and hierarchical transmission.

[0086] Secondly, abandoning the traditional learned vector quantization codebook, a branch-level finite scalar quantization strategy is adopted to discretely encode different frequency domain branches. High-precision quantization levels are configured for semantic branches to reduce semantic feature quantization distortion and ensure core semantic stability; low-precision quantization levels are configured for high-frequency detail branches to compress the transmission bitrate. By constraining the feature value range of each channel through learnable scaling parameters and a hyperbolic tangent activation function, a uniformly spaced implicit quantization grid is generated, eliminating the need to store embedded codebook tables. Combined with a pass-through estimator, backpropagation of training gradients is completed, completely removing redundant mechanisms such as commitment loss, codebook updates, and collapse repair.

[0087] Furthermore, a channel-aware, retraining-free progressive transmission scheduling mechanism is constructed, setting low-frequency semantic branches as fixed, mandatory core branches and high-frequency detail branches as schedulable, optional branches. The signal-to-noise ratio of the wireless channel is monitored in real time, and the number of high-frequency detail branches transmitted is dynamically adjusted according to a preset gradient threshold. When the channel quality is excellent, all detail branches are transmitted to improve image reconstruction accuracy; when the channel deteriorates, high-frequency detail branches are discarded step by step to reduce the transmission rate. A single model can adapt to multiple transmission rates without the need to retrain the model for different bandwidth scenarios.

[0088] Finally, the transmitting end sends the quantized discrete indexes of each transmission branch to the receiving end via an AWGN or Rayleigh fading digital channel; the receiving end dequantizes the indexes to restore the features of each frequency domain branch, and reconstructs the complete image by fusing them through inverse wavelet transform. Simultaneously, based on the principle of semantic classification invariance, the quantization distortion of semantic branches and the total error of channel disturbances are constrained to ensure that the semantic classification results of the image do not drift or become invalid under adverse channel conditions.

[0089] Compared to existing traditional vector quantization semantic communication technologies, the embodiments of this application possess several outstanding substantive features and significant advancements, greatly improving overall performance and engineering applicability. These are mainly reflected in the following aspects: First, the embodiments of this application achieve hierarchical isolation of semantic and detailed features through wavelet frequency domain decoupling, breaking the feature coupling bottleneck of traditional overall quantization and realizing a semantic-priority hierarchical protection mechanism. In harsh communication scenarios such as low signal-to-noise ratio and low bandwidth, the integrity and stability of core semantic information can be prioritized, achieving smooth gradient degradation. Compared with the traditional VQ scheme, the image semantic classification accuracy is improved by 5%–8%, the PSNR of the reconstructed image is improved by 2–4dB, and the LPIPS perceptual distortion is significantly reduced, greatly improving the communication robustness and task reliability under complex wireless channels.

[0090] Secondly, the embodiments of this application employ implicit finite scalar quantization instead of learned codebook quantization, eliminating the need to store large-scale codebook embedding tables. The synchronization metadata overhead at both ends is reduced from the traditional 500KB level to 0.03KB level, with synchronization overhead nearly zero. This completely solves the problems of high storage overhead and high synchronization costs in traditional solutions, perfectly adapting to the deployment of various resource-constrained lightweight wireless terminal devices. Furthermore, in the embodiments of this application, all quantization grid nodes are valid symbols, achieving 100% codebook activation utilization, completely eliminating the codebook collapse, symbol idleness, and resource waste problems commonly found in traditional VQ.

[0091] Third, the embodiments of this application significantly simplify the model training process, removing redundant mechanisms such as commitment loss, EMA codebook updates, and periodic codebook reinitialization required by traditional VQ, reducing training computational consumption and debugging difficulty, and improving model training stability and convergence speed. Simultaneously, the combination of a multi-weighted composite loss function and a three-stage course training strategy balances image pixel reconstruction accuracy, visual perception effect, semantic classification accuracy, and frequency domain feature consistency, resulting in superior overall model performance.

[0092] Fourth, the embodiments of this application realize single-model retraining and multi-rate adaptive transmission. By dynamically adding or removing high-frequency detail branches, it can adapt to multiple transmission bit rates. There is no need to train the model independently for different bandwidths and channel conditions, which greatly reduces the model iteration cost and engineering deployment cost. It effectively solves the problems of poor scenario adaptability and weak scalability of traditional models and can adapt to dynamically changing wireless channel environments.

[0093] Fifth, the embodiments of this application strictly control semantic quantization distortion error by configuring high-precision asymmetric quantization of semantic branches. Combined with semantic classification invariance constraints, sufficient channel disturbance error budget is reserved to effectively suppress semantic drift caused by noise and fading, ensuring the accuracy and stability of image semantic recognition under various channel conditions. It can be widely adapted to mainstream wireless channels such as AWGN and Rayleigh fading, and has extremely high versatility, practicality and industrialization value.

[0094] It should be noted that the semantic communication method provided in this application embodiment can be executed by a semantic communication device, or a control module within the semantic communication device for executing the loading semantic communication method. This application embodiment uses the execution of the loading semantic communication method by a semantic communication device as an example to illustrate the semantic communication method provided in this application embodiment.

[0095] Figure 6 This is a schematic diagram of the structure of an image semantic communication device with frequency domain hierarchical finite scalar quantization provided in an embodiment of this application. This image semantic communication device with frequency domain hierarchical finite scalar quantization is applied at the transmitting end, such as... Figure 6 As shown, the image semantic communication device with frequency domain hierarchical finite scalar quantization includes: The frequency domain decoupling module 610 is used to perform two-dimensional discrete wavelet transform frequency domain decomposition of the original image at the target level to obtain low-frequency semantic branches and multi-scale high-frequency detail branches. The finite scalar quantization module 620 is used to perform finite scalar quantization processing on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively through the semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index. The transmission scheduling module 630 is used to select the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index according to the channel quality parameters. The transmission module 640 is used to transmit the low-frequency quantization discrete index and the target high-frequency quantization discrete index to the receiving end via a wireless channel.

[0096] Optionally, the semantic communication model includes: a first branch encoder and a first finite scalar quantization module corresponding to the low-frequency semantic branch, and a second branch encoder and a second finite scalar quantization module corresponding to the high-frequency detail branches at each scale, respectively. The finite scalar quantization module includes: The low-frequency coding unit is used to encode the low-frequency semantic branch through the first branch encoder to obtain low-frequency hidden layer features; A high-frequency coding unit is used to encode the high-frequency detail branch for each scale using a second branch encoder corresponding to the scale, thereby obtaining high-frequency hidden layer features. The low-frequency quantization unit is used to perform finite scalar quantization processing on the low-frequency hidden layer features according to the first quantization level corresponding to the low-frequency semantic branch through the first finite scalar quantization module to obtain the low-frequency quantized discrete index. The high-frequency quantization unit is used to perform finite scalar quantization on the high-frequency hidden layer features of each scale according to the second quantization level corresponding to the high-frequency detail branch through the second finite scalar quantization module corresponding to the scale, so as to obtain the high-frequency quantized discrete index of the scale. The first quantization level is higher than the second quantization level.

[0097] Optionally, the low-frequency quantization unit is specifically used for: The first finite scalar quantization module determines the first quantization interval based on the first quantization level, and performs a rounding operation on the low-frequency hidden layer features based on the first quantization interval and the first feature channel value range constrained by the network parameters of the first finite scalar quantization module to obtain the low-frequency quantized discrete index.

[0098] Optionally, the high-frequency quantization unit is specifically used for: For each high-frequency detail branch at each scale, the second quantization interval is determined by the second finite scalar quantization module corresponding to the scale based on the second quantization level. Then, based on the second quantization interval and the range of values ​​of the second feature channel constrained by the network parameters of the second finite scalar quantization module, the high-frequency hidden layer features at the scale are rounded to obtain the high-frequency quantized discrete index of the scale.

[0099] Optionally, the semantic communication model further includes: a finite scalar dequantization module and a decoder located at the receiving end; The training process of the semantic communication model includes: In the absence of a channel, the initial semantic communication model is pre-trained based on training samples to obtain a pre-trained semantic communication model. With the addition of a wireless channel, the pre-trained semantic communication model is trained based on the training samples, and all high-frequency detail branches are transmitted during the training process to obtain an intermediate semantic communication model. With the addition of a wireless channel, the intermediate semantic communication model is trained based on the training samples, and during the training process, the high-frequency detail branch to be transmitted is selected according to the signal-to-noise ratio of the wireless channel to obtain the trained semantic communication model.

[0100] Optionally, the semantic communication model employs a composite loss function during training, which includes a pixel reconstruction loss function, a visual perception loss function, a semantic classification cross-entropy loss function, and a frequency domain consistency loss function.

[0101] Optionally, the channel quality parameters include the signal-to-noise ratio; The transmission scheduling module is specifically used for: The signal-to-noise ratio is compared with a plurality of gradient-incrementing signal-to-noise ratio thresholds to determine the target number of signal-to-noise ratio thresholds smaller than the signal-to-noise ratio, wherein the total number of gradient-incrementing signal-to-noise ratio thresholds is the same as the target level. According to the transmission priority from high to low, the target number of high-frequency quantization discrete indices are selected from the multi-scale high-frequency quantization discrete indices as the target high-frequency quantization discrete indices.

[0102] Optionally, the two-dimensional discrete wavelet transform includes the Haar wavelet transform; The frequency domain decoupling module is specifically used for: The original image is subjected to frequency domain decomposition using the target level Haar wavelet transform to obtain one approximate sub-band and three directional detail sub-bands corresponding to each level; The final-level approximate subband is used as the low-frequency semantic branch; The three-way detail subbands of each level are spliced ​​together to obtain the high-frequency detail branches of the corresponding scale.

[0103] The image semantic communication device with frequency domain hierarchical finite scalar quantization in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0104] The image semantic communication device with frequency domain hierarchical finite scalar quantization in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0105] The image semantic communication device with frequency domain hierarchical finite scalar quantization provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0106] The image semantic communication device with frequency-domain hierarchical finite scalar quantization provided in this application decomposes the original image into a low-frequency semantic branch and a multi-scale high-frequency detail branch by performing a two-dimensional discrete wavelet transform frequency domain decomposition on the original image at the target level. This physically decouples semantic information from detail information, providing a foundation for differentiated protection and hierarchical transmission. The device performs finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively using a semantic communication model, resulting in a low-frequency quantized discrete index and a multi-scale high-frequency quantized discrete index. Using finite scalar quantization, the transmitting and receiving ends do not need to synchronously store codebook embedding tables, saving resources. Furthermore, the device selects the target high-frequency quantized discrete index to be transmitted from the multi-scale high-frequency quantized discrete index based on channel quality parameters, and transmits both the low-frequency quantized discrete index and the target high-frequency discrete index to the receiving end via a wireless channel. This allows for selection of the corresponding semantic branch for transmission based on channel quality. Even in scenarios with poor channel quality, priority can be given to ensuring the normal transmission of the low-frequency quantized discrete index representing the core semantics, improving the robustness of communication transmission.

[0107] Figure 7 This is a schematic diagram of the structure of an image semantic communication device with frequency domain hierarchical finite scalar quantization provided in an embodiment of this application. This image semantic communication device with frequency domain hierarchical finite scalar quantization is applied at the receiving end, such as... Figure 7 As shown, the image semantic communication device with frequency domain hierarchical finite scalar quantization includes: The receiving module 710 is used to receive the quantized discrete index sent by the sending end. The quantized discrete index includes the low-frequency quantized discrete index corresponding to the low-frequency semantic branch and the target high-frequency quantized discrete index corresponding to the target high-frequency detail branch. The dequantization module 720 is used to perform finite scalar dequantization processing on the low-frequency quantized discrete index and the target high-frequency quantized discrete index through a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The image reconstruction module 730 is used to perform inverse wavelet transform processing on the low-frequency semantic branch and the target high-frequency detail branch to obtain the reconstructed image of the original image.

[0108] Optionally, the semantic communication model includes a finite scalar dequantization module and a reconstruction decoder. The finite scalar dequantization module includes a first finite scalar dequantization module corresponding to the low-frequency semantic branch and a second finite scalar dequantization module corresponding to the high-frequency detail branches at each scale. The dequantization module is specifically used for: The low-frequency quantized discrete index is dequantized using finite scalar dequantization through the first finite scalar dequantization module to obtain the restored low-frequency hidden layer features. The target high-frequency quantized discrete index is dequantized by the second finite scalar dequantization module corresponding to the target high-frequency detail branch to obtain the restored target high-frequency hidden layer features. The reconstruction decoder decodes the restored low-frequency hidden layer features and the restored target high-frequency hidden layer features to obtain the low-frequency semantic branch and the target high-frequency detail branch.

[0109] Optionally, the semantic communication model further includes a semantic decoder and a classifier; The device further includes: The semantic decoding module is used to decode the low-frequency hidden layer features through the semantic decoder to obtain semantic features; The semantic verification module is used to classify the semantic features as semantically valid or semantically invalid using the classifier, and obtain the classification result.

[0110] The image semantic communication device with frequency domain hierarchical finite scalar quantization in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0111] The image semantic communication device with frequency domain hierarchical finite scalar quantization in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0112] The image semantic communication device with frequency domain hierarchical finite scalar quantization provided in this application embodiment can achieve... Figure 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0113] The image semantic communication device with frequency domain hierarchical finite scalar quantization provided in this application receives a quantized discrete index sent by the transmitting end. The quantized discrete index includes a low-frequency quantized discrete index corresponding to a low-frequency semantic branch and a target high-frequency quantized discrete index corresponding to a target high-frequency detail branch. The low-frequency quantized discrete index and the target high-frequency quantized discrete index are dequantized using a semantic communication model with finite scalar quantization to obtain the low-frequency semantic branch and the target high-frequency detail branch. The low-frequency semantic branch and the target high-frequency detail branch are then subjected to inverse wavelet transform processing to obtain the reconstructed image of the original image. By using finite scalar quantization, the transmitting and receiving ends do not need to synchronously store the codebook embedding table, which can save resource consumption. Moreover, the target high-frequency quantized discrete index is a portion of the high-frequency quantized discrete index selected by the transmitting end from the multi-scale high-frequency detail branches corresponding to the multi-scale quantized discrete index obtained by wavelet transform decomposition based on channel quality parameters. In this way, the corresponding semantic branch can be selected for transmission based on channel quality. In scenarios with poor channel quality, the low-frequency quantized discrete index representing the core semantics can be transmitted normally first, which can improve the robustness of communication transmission.

[0114] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described frequency-domain hierarchical finite scalar quantization image semantic communication method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0115] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0116] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0117] The electronic device 800 includes, but is not limited to, components such as: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810. The input unit 804 may include an image processor 8041 and a microphone 8042; the display unit 806 may include a display panel 8061; and the user input unit 807 may include a touch panel 8071 and other input devices (such as a keyboard) 8072.

[0118] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0119] The memory 809 stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor 810, it implements the various processes of the above-described frequency-domain hierarchical finite scalar quantization image semantic communication method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0120] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described frequency-domain hierarchical finite scalar quantization image semantic communication method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0121] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0122] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described semantic communication method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0123] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0124] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0126] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A frequency-domain hierarchical finite scalar quantization method for image semantic communication, characterized in that, Applied to the sending end, including: The original image is subjected to two-dimensional discrete wavelet transform frequency domain decomposition at the target level to obtain low-frequency semantic branches and multi-scale high-frequency detail branches; The low-frequency semantic branch and the multi-scale high-frequency detail branch are processed by finite scalar quantization using a semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index. Based on the channel quality parameters, the target high-frequency quantization discrete index to be transmitted is selected from the multi-scale high-frequency quantization discrete index. The low-frequency quantization discrete index and the target high-frequency quantization discrete index are transmitted to the receiving end via a wireless channel.

2. The method according to claim 1, characterized in that, The semantic communication model includes: a first branch encoder and a first finite scalar quantization module corresponding to the low-frequency semantic branch, and a second branch encoder and a second finite scalar quantization module corresponding to the high-frequency detail branches at each scale, respectively. The step of performing finite scalar quantization on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index includes: The low-frequency semantic branch is encoded by the first branch encoder to obtain low-frequency hidden layer features; For each high-frequency detail branch at each scale, the high-frequency detail branch is encoded by the second branch encoder corresponding to the scale to obtain high-frequency hidden layer features; The first finite scalar quantization module determines the first quantization interval based on the first quantization level corresponding to the low-frequency semantic branch, and performs a rounding operation on the low-frequency hidden layer features based on the first quantization interval and the first feature channel value range constrained by the network parameters of the first finite scalar quantization module to obtain the low-frequency quantized discrete index. For each high-frequency detail branch at each scale, the second finite scalar quantization module corresponding to the scale determines the second quantization interval based on the second quantization level corresponding to the high-frequency detail branch. Then, based on the second quantization interval and the range of values ​​of the second feature channel constrained by the network parameters of the second finite scalar quantization module, the high-frequency hidden layer features at the scale are rounded to obtain the high-frequency quantized discrete index of the scale. The first quantization level is higher than the second quantization level.

3. The method according to claim 2, characterized in that, The semantic communication model also includes: a finite scalar dequantization module and a decoder located at the receiving end; The training process of the semantic communication model includes: In the absence of a channel, the initial semantic communication model is pre-trained based on training samples to obtain a pre-trained semantic communication model. With the addition of a wireless channel, the pre-trained semantic communication model is trained based on the training samples, and all high-frequency detail branches are transmitted during the training process to obtain an intermediate semantic communication model. With the addition of a wireless channel, the intermediate semantic communication model is trained based on the training samples, and during the training process, the high-frequency detail branch to be transmitted is selected according to the signal-to-noise ratio of the wireless channel to obtain the trained semantic communication model.

4. The method according to any one of claims 1-3, characterized in that, The channel quality parameters include the signal-to-noise ratio; The step of selecting the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index based on channel quality parameters includes: The signal-to-noise ratio is compared with a plurality of gradient-incrementing signal-to-noise ratio thresholds to determine the target number of signal-to-noise ratio thresholds smaller than the signal-to-noise ratio, wherein the total number of gradient-incrementing signal-to-noise ratio thresholds is the same as the target level. According to the transmission priority from high to low, the target number of high-frequency quantization discrete indices are selected from the multi-scale high-frequency quantization discrete indices as the target high-frequency quantization discrete indices.

5. A frequency-domain hierarchical finite scalar quantization method for image semantic communication, characterized in that, Applied to the receiving end, including: The receiver sends a quantized discrete index, which includes a low-frequency quantized discrete index corresponding to a low-frequency semantic branch and a target high-frequency quantized discrete index corresponding to a target high-frequency detail branch. The low-frequency quantized discrete index and the target high-frequency quantized discrete index are dequantized using a finite scalar model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The low-frequency semantic branch and the target high-frequency detail branch are subjected to inverse wavelet transform processing to obtain the reconstructed image of the original image.

6. The method according to claim 5, characterized in that, The semantic communication model includes a finite scalar dequantization module and a reconstruction decoder. The finite scalar dequantization module includes a first finite scalar dequantization module corresponding to the low-frequency semantic branch and a second finite scalar dequantization module corresponding to the high-frequency detail branches at each scale. The step of performing finite scalar dequantization processing on the low-frequency quantized discrete index and the target high-frequency quantized discrete index through a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch includes: The low-frequency quantized discrete index is dequantized using the first finite scalar dequantization module to obtain the restored low-frequency hidden layer features. The target high-frequency quantized discrete index is dequantized by a second finite scalar dequantization module corresponding to the target high-frequency detail branch to obtain the restored target high-frequency hidden layer features. The reconstruction decoder decodes the restored low-frequency hidden layer features and the restored target high-frequency hidden layer features to obtain the low-frequency semantic branch and the target high-frequency detail branch.

7. An image semantic communication device with frequency domain hierarchical finite scalar quantization, characterized in that, Applied to the sending end, including: The frequency domain decoupling module is used to perform target-level two-dimensional discrete wavelet transform frequency domain decomposition on the original image to obtain low-frequency semantic branches and multi-scale high-frequency detail branches. The finite scalar quantization module is used to perform finite scalar quantization processing on the low-frequency semantic branch and the multi-scale high-frequency detail branch respectively through the semantic communication model to obtain the low-frequency quantized discrete index and the multi-scale high-frequency quantized discrete index. The transmission scheduling module is used to select the target high-frequency quantization discrete index to be transmitted from the multi-scale high-frequency quantization discrete index according to the channel quality parameters. The transmission module is used to transmit the low-frequency quantization discrete index and the target high-frequency quantization discrete index to the receiving end via a wireless channel.

8. An image semantic communication device with frequency-domain hierarchical finite scalar quantization, characterized in that, Applied to the receiving end, including: The receiving module is used to receive the quantized discrete index sent by the sending end. The quantized discrete index includes the low-frequency quantized discrete index corresponding to the low-frequency semantic branch and the target high-frequency quantized discrete index corresponding to the target high-frequency detail branch. The dequantization module is used to perform finite scalar dequantization processing on the low-frequency quantized discrete index and the target high-frequency quantized discrete index through a semantic communication model to obtain the low-frequency semantic branch and the target high-frequency detail branch. The image reconstruction module is used to perform inverse wavelet transform processing on the low-frequency semantic branch and the target high-frequency detail branch to obtain the reconstructed image of the original image.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the image semantic communication method of frequency domain hierarchical finite scalar quantization as described in any one of claims 1-4 or 5-6.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image semantic communication method of frequency domain hierarchical finite scalar quantization as described in any one of claims 1-4 or 5-6.

Images