Lightweight image semantic communication method and system and readable storage medium

Abstract

The invention relates to the technical field of semantic communication, and discloses a lightweight image semantic communication method and system and a readable storage medium, and the method comprises the steps: a sending end obtains a source image, employs a deep learning model to extract the texture semantics of the source image, and employs a traditional visual algorithm to extract the color semantics of the source image; the sending end sends the color semantics and the texture semantics to a receiving end; and the receiving end fuses the color semantics and the texture semantics and reconstructs the source image. According to the method, a semantic communication task is split, a simple traditional computer vision algorithm is introduced, a part of deep learning models used in the semantic communication process in the prior art are replaced, significant high delay possibly caused by excessive dependence on the deep models is avoided, and the problem that in the prior art, a semantic communication method completely depends on a neural network model, and the semantic communication efficiency is high is solved. Therefore, the problem of difficulty in real machine deployment and use in the whole scene under the existing hardware condition is solved.

Description

Technical Field

[0001] The present invention relates to the field of semantic communication technology, and in particular to a lightweight image semantic communication method, system and readable storage medium. Background Art

[0002] In the field of image semantic communication, existing research has shown that deep learning models can replace most of the original codecs and achieve better compression and accuracy performance than traditional methods. Therefore, the research on source-channel joint coding based on deep learning has greatly promoted the research on semantic communication systems. Almost all existing semantic communication schemes use the form of source-channel joint coding.

[0003] However, even on GPU servers with high computing power, almost all of these better semantic communication algorithms based on deep learning source-channel joint coding require at least one hundred or even thousands of milliseconds for model initialization and model inference. This level of delay is unacceptable when developing real-world communication systems that are sensitive to latency. Especially considering that a large part of the communication will occur on edge devices, which usually have lower computing power and limited memory in addition to latency issues. Therefore, one of the biggest challenges in applying deep learning-based semantic coding algorithms on such edge devices is that they may not be able to provide enough computing power to complete the heavy calculation of model inference in a short time. In actual applications, the time it takes to complete these huge semantic encoding and decoding based on deep learning may even be longer than the time it takes to complete the entire communication process using traditional communication algorithms, which is unreasonable in most application scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a lightweight image semantic communication method, system and readable storage medium, aiming to solve the problem that the semantic communication method in the prior art requires too much computing power and is difficult to use under existing hardware conditions.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows: The present invention provides a lightweight image semantic communication method, the lightweight image semantic communication method comprising: The sending end obtains a source image, uses a deep learning model to extract the texture semantics of the source image, and uses a traditional visual algorithm to extract the color semantics of the source image; The sending end sends the color semantics and the texture semantics to the receiving end; The receiving end fuses the color semantics and the texture semantics to reconstruct the source image.

[0006] Further, the sending end sends the color semantics and the texture semantics to the receiving end, specifically including: The sending end quantizes the color semantics into quantized color semantics and quantizes the texture semantics into quantized texture semantics; The sending end transmits the quantized color semantics and the quantized texture semantics to the receiving end; The receiving end uses a traditional visual algorithm to restore the quantized color semantics to obtain the color semantics, and uses a texture restoration model to process the quantized texture semantics to restore the texture semantics.

[0007] Furthermore, the transmitting end and the receiving end both include a plurality of auxiliary devices, the deep learning model includes a texture extraction model and a shallow neural network, and the extraction of texture semantics of the source image using the deep learning model specifically includes: Inputting the source image into a shallow neural network to extract shallow features of the source image; Dividing the shallow features into a plurality of feature blocks; A number of the feature blocks are distributed to a plurality of auxiliary devices, and each auxiliary device processes the feature blocks respectively using a texture extraction model to obtain the texture semantics.

[0008] Further, the adopting a texture restoration model to process the quantized texture semantic restoration to obtain the texture semantics specifically includes: Semantically segmenting the quantized texture into a plurality of semantic blocks; The plurality of semantic blocks are distributed to a plurality of auxiliary devices, and each auxiliary device processes the semantic block respectively by using a texture restoration model to restore the texture semantics.

[0009] Furthermore, dividing the shallow features into a plurality of feature blocks specifically includes: Get the segmentation consumption model, and calculate the segmentation size of each auxiliary device based on the segmentation consumption model with the goal of minimizing the total time consumption: ; ; ; ; in, is the total time consumed, Indicates constraints, Indicates The memory consumption of each auxiliary device at the corresponding partition size, Indicates The maximum amount of memory available to auxiliary devices. is the maximum number of available devices, is the number of auxiliary devices; According to each segmentation size, the shallow feature is segmented into a plurality of feature blocks; The semantic segmentation of the quantized texture into a plurality of semantic blocks specifically includes: Get the segmentation consumption model, and calculate the segmentation size of each auxiliary device based on the segmentation consumption model with the goal of minimizing the total time consumption: ; ; ; ; According to each of the segmentation sizes, the quantized texture is semantically segmented into a plurality of semantic blocks.

[0010] Furthermore, the obtaining of the segmentation consumption model specifically includes: Establish a parallel time-consuming model under the real protocol of homogeneous systems: ; in, The actual protocol parallel time of the homogeneous system, The number of CPU cycles required to complete the inference calculation for a unit of input data, is the transmission data rate between the receiving end and the auxiliary device, It is the data of auxiliary equipment rate, Indicates the number of CPU cycles that the auxiliary device can complete per unit time. is the size of the additional data to be transmitted. To quantify the semantic size of texture, The number of assistive devices available; Establish a parallel time-consuming model under the real protocol of heterogeneous systems: ; in, represents the actual protocol parallel time of the heterogeneous system, Representative auxiliary equipment The size of the data block to be processed, Represents the primary and secondary devices The transmission rate between Representative auxiliary equipment Data rate, Representative auxiliary equipment The number of CPU cycles that can be completed per unit time; Model memory consumption of auxiliary devices: ; in, The memory required to complete the inference calculation of the unit input data, The memory required to load the model parameter weights; The parallel time consumption model under the real protocol of the homogeneous system, the memory consumption model and the set of the parallel time consumption model under the real protocol of the heterogeneous system are used as the consumption model.

[0011] Furthermore, the receiving end fuses the color semantics and the texture semantics to reconstruct the source image, which specifically includes: Inputting the color semantics and the texture semantics into a fusion model; The fusion model reconstructs and outputs the source image according to the color semantics and the texture semantics.

[0012] Furthermore, the fusion model, the texture extraction model and the texture restoration model are obtained through training, and the loss function of the training is: ; in, represents the loss value, represents the source image, represents the reconstructed image, and To train the hyperparameters, represents the texture extraction operator, represents the texture distribution of the source image, Represents the texture distribution of the reconstructed image.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also provides a lightweight semantic communication system, the lightweight semantic communication system comprising: a sending end and a receiving end, the sending end and the receiving end establishing a communication connection; The sending end is used to obtain a source image, extract texture semantics of the source image using a neural network, extract color semantics of the source image using a traditional visual algorithm, and send the color semantics and the texture semantics to the receiving end; The receiving end is used to fuse the color semantics and the texture semantics to reconstruct the source image.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a readable storage medium, which stores a lightweight image semantic communication program, and when the lightweight image semantic communication program is executed by a processor, it implements the steps of the lightweight image semantic communication method as described above.

[0015] The present invention adopts the above technical solution to achieve the following effects: The present invention splits the semantic communication task and introduces a simple traditional computer vision algorithm to replace some deep learning models used in the semantic communication process in the prior art. It simplifies the goals that deep learning needs to achieve and avoids the significant high latency that may be caused by excessive use of deep models. It solves the problem that the semantic communication method in the prior art relies entirely on the neural network model, resulting in excessive computing power required and difficulty in real machine deployment and use in all scenarios under existing hardware conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of the steps of a lightweight image semantic communication method in a preferred embodiment of the present invention; Figure 2 It is a full flow chart of a lightweight image semantic communication method in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the relationship between the number of devices and time consumption in a preferred embodiment of the present invention; Figure 4 Schematic diagram of calculation of texture information entropy and texture relative entropy in a preferred embodiment of the present invention; Figure 5 A schematic diagram of the operating environment of a preferred embodiment of the system of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Embodiment 1 The present invention is used Represents the input source image, the sender uses the encoding function The source image Mapped to some finite number of complex symbols , , Indicates plural, Represents a complex set The first complex symbol in Represents a complex set The second complex symbol in Represents a complex set The plural symbols, Represents a complex set The number of elements, this process can be expressed as , then, the encoded symbols are transmitted through the channel, which is described as .in, is the channel gain, is Gaussian noise. Finally, the receiver applies the decoding function To reconstruct the source image and obtain the reconstructed image , this process is described as .

[0019] A common strategy for building a JSCC (Joint source-channel coding) communication system based on deep learning is to train an autoencoder and use its encoder and decoder as the encoding and decoding functions of the communication system, which greatly simplifies the construction process of the communication system because only the autoencoder needs to be trained end-to-end. However, greedily using deep models may lead to significantly high latency. Therefore, the goal of the present invention is to replace some deep models with some efficient traditional computer vision algorithms to solve the high latency problem.

[0020] Specifically, see Figure 1 and Figure 2 , Embodiment 1 of the present application is a lightweight image semantic communication method, which comprises the steps of: S1. The sending end obtains a source image, uses a deep learning model to extract the texture semantics of the source image, and uses a traditional visual algorithm to extract the color semantics of the source image.

[0021] A key challenge in the practical development of deep semantic communication systems is the extremely limited edge resource constraints. Therefore, the complexity of the method must be minimized. In computer vision, some efficient traditional algorithms can effectively complete simple tasks, so the present invention partially replaces some deep learning models with some traditional algorithms.

[0022] Specifically, in this embodiment, two different processing methods are used for the source image to extract and restore its color and texture semantics respectively, and finally, the source image is reconstructed by fusing the color and texture semantics.

[0023] In this embodiment, the color semantics is extracted by using a Gaussian filter. Gaussian downsampling can smooth the image and remove high-frequency information. Since color semantics belongs to low-frequency information, the filter will retain this information. Therefore, for color semantics, the extraction and reconstruction steps can be composed of Gaussian pyramid downsampling and upsampling. By using a simple traditional computer vision algorithm, color semantics can be extracted with extremely low resource consumption. In step S1, only the extraction step is involved, and the process can be described as: ; in, is the extracted color semantics, is Gaussian downsampling.

[0024] For texture semantics, this embodiment trains a learning algorithm with learnable parameters The autoencoder is used as the texture extraction model. The texture extraction model uses dense blocks to process the features of each stage: ; in, is the texture extraction model, is the extracted texture semantics.

[0025] S2. The sending end sends the color semantics and the texture semantics to the receiving end.

[0026] After that, the transmitter sends the texture semantics and color semantics to the receiver. Specifically, the texture semantics and color semantics are first quantized and encoded respectively: ; ; in, It is a quantitative operation. is the quantized texture semantics, It is the quantized color semantics. The quantized texture semantics and color semantics are transmitted to the receiving end. The receiving end restores the color semantics and texture semantics according to the quantized texture semantics and color semantics. The whole process can be described as: ; ; in, is the signal-to-noise ratio (SNR), is the communication channel, is Gaussian upsampling, is the texture restoration model, which is a decoding function.

[0027] In an optional embodiment, in order to deploy and accelerate the lightweight image semantic communication method of this embodiment on a single edge device or edge cluster with extremely limited computing resources, the present invention adds a feature segmentation step before the first stage texture processing, and divides and distributes the computing tasks to other limited devices in the local network / cluster for parallel computing, or divides and calculates serially on the current device. Experiments show that the method of the present invention can complete the computing tasks directly on the edge device after segmentation without reducing the transmission quality, while other previous methods cannot be effectively completed on the edge device.

[0028] Specifically, those skilled in the art know that the first difficulty usually encountered when deploying deep models on edge devices is the high inference latency due to the low computing power of the device. Addressing this weakness is critical to designing a robust communication system, and it is not always solved by reducing the size of the deep network, as this is likely to result in a very significant decrease in communication accuracy. In addition to low computing power, memory limitations are also a typical problem for edge devices. The memory consumed by processing visual data using deep models often exceeds the maximum available memory of edge devices, which greatly limits the deployment scenarios of existing deep learning-based semantic communication methods. To alleviate this problem, designing an edge deployment framework with joint perception-communication-computing capabilities may be a relatively good choice to accelerate real-world deep semantic communication systems.

[0029] Because the cost of upgrading / replacing the entire device is much higher and the flexibility is much lower, considering the cost issue, it may not always be possible to accept such a solution. In fact, in the idea of ​​edge computing technology, if the inference calculation can be divided into several parts and executed in parallel on multiple devices in a cluster or collaborative network, it can better solve this problem. Because the devices in the edge network are not always busy, they can share some computing resources with other devices in the same sub-network. In addition, a weak edge device can be connected to several weak computing nodes for a simple and low-cost computing power upgrade instead of replacing the entire device.

[0030] Therefore, in this embodiment, before texture extraction, the source image is first input into a shallow neural network to extract shallow features of the source image, and then the shallow features are segmented into a number of feature blocks, and the feature blocks are assigned to a number of auxiliary devices. Each auxiliary device uses a texture extraction model to process the feature blocks respectively to obtain texture semantics. Before texture restoration, the quantized texture semantics is segmented into a number of semantic blocks, and the semantic blocks are assigned to a number of auxiliary devices. Each auxiliary device uses a texture restoration model to process the semantic blocks respectively to restore the texture semantics. This is to achieve segmentation processing of inference calculations for texture extraction and texture restoration.

[0031] Then, we need to consider how to split the inference calculation into several blocks and redistribute them to different devices to complete the calculation without reducing the accuracy. The above problem has been considered in the previous proposal and an appropriate solution has been provided. The semantic block segmentation step in the proposed feature processing can split the inference calculation into several blocks to achieve redistribution and distributed computing on heterogeneous devices, or complete small-scale calculations on a single device in multiple times to significantly reduce the consumption of memory resources.

[0032] In this embodiment, the inference delay of the semantic communication codec based on the convolutional neural network is: ; in, To quantify the semantic size of texture, represents the inference delay, Represents each convolutional layer, represents the total number of convolutional layers, and Respectively represent The width and height of the layer input feature map, Indicates The convolution kernel size of the layer, and Respectively represent the number of input and output channels of the layer, Indicates the number of CPU cycles that the device performing inference calculation can complete per unit time. Since the convolutional neural network model is fixed in the inference stage, the number of CPU cycles required for each unit input is also fixed. If the number of CPU cycles required for each unit input is set to , it can be seen that the inference delay and quantized texture semantic size There is a linear relationship.

[0033] First consider the isomorphic scenario and use represents the number of auxiliary devices. Assuming that semantic blocks are transmitted to each device in a linear rather than parallel manner, and an ideal protocol is used to establish a connection without the need for additional transmission of data (such as headers), the time taken is: ; in, The ideal protocol for homogeneous systems takes time, is the data rate transmitted between the master and slave devices, It is the data of auxiliary equipment rate. It can be found that the time complexity of this process is ,like Figure 3 As shown, the time consumption and the number of auxiliary equipment Inversely proportional to the number of auxiliary devices Approaches infinity, and the time consumption approaches a constant. However, actual protocols are usually not like ideal protocols. They usually need to waste some communication resources to transmit additional data (such as headers). In this case, the time consumption is: ; in, is the actual protocol time of the homogeneous system, and the time complexity is , is the additional data size, initial time, and number of auxiliary devices is inversely proportional to the size of the semantic block data because the size of the header is relatively small and can be ignored. As it approaches infinity, the time complexity approaches .

[0034] If semantic blocks are transmitted in parallel instead of linearly, the time required to use the above ideal protocol or actual protocol will become: ; ; in, The ideal protocol parallel time for homogeneous systems, is the actual protocol parallel time of the homogeneous system, and their time complexity is , the delay decreases in inverse proportion. In fact, as long as the number of devices is limited, parallel transmission is a common strategy in practice.

[0035] Based on this, we can try to extend the formula to a heterogeneous scenario. In this case, the total delay is equal to the maximum time consumed by all devices. The calculation formula is: ; in, represents the actual protocol parallel time of the heterogeneous system, Representative Available auxiliary equipment, is the total number of auxiliary devices, Represented by auxiliary equipment The size of the data processed, Represents the primary and secondary devices The transmission rate between Representative auxiliary equipment Data rate, Representative auxiliary equipment The number of CPU cycles that can be completed per unit time.

[0036] In this embodiment, another important factor to consider is the memory consumption of device reasoning, the memory consumption of the semantic communication codec based on the convolutional neural network for: ; When the convolutional neural network model is fixed, the memory occupied by the parameter weights and the memory required for the inference operation of each unit input are fixed. Set the memory occupied by the parameter weights to , the memory required for the inference calculation of unit input data is set to , then the formula can be further simplified to a linear relationship. In heterogeneous, auxiliary equipment Memory consumption for: ; When designing a semantic communication system, we always hope that the total transmission delay is as low as possible, while the memory consumption of the system on each device has an upper limit that the device can support. Therefore, the objective function of the entire acceleration process can be described as: ; ; ; ; in, is the total time consumption, and this objective function can be used to find the data size , so that the total time consumption is minimized, while the memory consumption of each device Does not exceed the upper limit of available memory . Is the maximum number of devices available in the local network or cluster.

[0037] In a homogeneous system, In heterogeneous systems, In both cases, solving the optimization problem requires knowing the amount of computation required to compute each unit of input, the maximum available computing power and memory of each edge device in the cluster, and the and the transmission rate with the master device. Finally, use the corresponding The objective function of the above acceleration process is solved based on the available resource information to determine the block computing strategy.

[0038] S3. The receiving end fuses the color semantics and the texture semantics to reconstruct the source image.

[0039] The receiver uses a learnable parameter The deep network block of is used as a fusion model to fuse color and texture semantics to restore the source image: ; in, is the reconstructed image, is the fusion function of fusion semantics.

[0040] The present invention uses a loss function to train a texture restoration model, a texture processing model, and a deep network block. Specifically, in an optional embodiment, during the training of the texture processing model, the texture restoration model, and the deep network block, the loss function used in this embodiment is a loss function of a combination of mean square error and SSIM (Mean Structural Similarity Index Measure, structural similarity), which is as follows: ; in, represents the loss value, are two hyperparameters, represents a real number, represents the structural similarity function.

[0041] During the training process, this embodiment uses the Adam optimizer to determine the minimum distortion , and value: ; in, is the joint probability distribution of the source image and the reconstructed image, Denotes the joint probability distribution The expected value of express Parameters, express Parameters, express Parameters.

[0042] Finally, the present invention evaluates the way in which the method of the present invention performs semantic communication. In practice, the present invention finds that images of similar quality judged by classic metrics such as mean square error (MSE) and peak signal-to-noise ratio (PSNR) do not always match the quality perceived by humans subjectively.

[0043] This is because these metrics usually focus on the low-frequency information of the image (such as color) and ignore the high-frequency information (such as details and texture). Therefore, such metrics may not accurately show the distortion of all types of information. In addition, since the human visual system is not sensitive to subtle color differences, these metrics cannot always correctly approximate human perception, especially when the image quality is relatively high. To alleviate this problem, some other metrics have been proposed in the prior art, such as perceptual loss and adversarial loss. Perceptual loss uses a pre-trained deep neural network (such as VGG-16) and assumes that it has the ability to perceive semantic information, extracts important features, and then measures the distortion of the image from the perspective of extracted features. Adversarial loss uses a discriminator based on a deep neural network to check whether the images have similar probability distributions. Unfortunately, adversarial loss is often unstable during training, and both of the above losses have the problem of low interpretability of deep neural networks. They are not a good way to measure the distortion of high-resolution images. Therefore, a better way to measure quality and explain the difference from the perspective of evaluation and human perception is needed.

[0044] Specifically, the indicators proposed in this invention need to think about what helps humans get information from images. In daily life, it can be found that humans are good at getting a lot of information from pictures composed of lines, such as sketches. These lines describe the basic structure of the image and provide the main information of the image. Based on this phenomenon, some lines can be used to describe the information of the image. More precisely, the texture of the image can provide humans with information about the image. Secondly, a method to measure information is needed. Since the microscopic displacement of texture lines usually does not affect the information, distortion metrics that are strictly pixel-based, such as mean absolute error (MAE) or mean square error (MSE), should not be used.

[0045] In fact, previous information theory research has provided some valuable tools for measuring information. Information entropy is a powerful tool to measure the amount of information from a given source. Figure 4 As shown in (a) in the figure, the information entropy of the image texture can be calculated to approximately determine the amount of information that can be obtained by the human perspective. Specifically, the original image is used as the standard to check whether the amount of information provided by the reconstructed image is close. However, although texture information entropy (TIE) can accurately describe the amount of information, it does not always show the quality of the information. If one image contains a lot of erroneous and confusing information, and the other image contains limited but accurate information, then the first image may have a higher TIE value, but the information quality is worse. This weakness will strictly limit the use of this measurement method.

[0046] Since the evaluation of information entropy only focuses on a single target, it is difficult to give the relationship between images. In contrast, relative entropy (also known as Kullback-Leibler (KL) divergence or information divergence) can well describe the divergence of information distribution. Please refer to Figure 4 (b) in the figure shows the process of determining the texture relative entropy (TRE). First, an edge detection algorithm is used to extract texture from the image. Specifically, the present invention uses the Canny operator as the edge detection algorithm. Next, a sliding window is used to intercept small texture blocks. The size of each small block intercepted by the sliding window is determined by a hyperparameter OK, this parameter determines the pixel displacement allowed for the texture. The range is from 1 to the size of the image. In order to obtain more representative evaluation results, this embodiment selects the one that can maximize the standard deviation of the evaluation results of various reconstruction methods. value.

[0047] Finally, the average relative entropy of the corresponding texture block is calculated to determine the divergence of the local information distribution. This process can be expressed as: .

[0048] in, and is the height and width of the original / reconstructed image, and are the height and width of the sliding window, Represents the local information distribution of the texture block of the original image cut out by the sliding window, Represents the local information distribution of the texture block of the reconstructed image cut out by the sliding window, represents the KL divergence, Represents the texture relative entropy.

[0049] Among these parameters, , , and can be regarded as constants. and Represent the texture distribution of the original image and the reconstructed image respectively. If and If the original image and the reconstructed image are close, then they are similar. To minimize information distortion, just make and The gap between them can be minimized. It is not difficult to see that when and When the values ​​are the same, the maximum lower bound of TRE is 0 and there is no upper bound. In addition, the above TRE calculation is asymmetric because the KL divergence is asymmetric. TRE can be easily made symmetric by replacing the KL divergence with the Jensen-Shannon (JS) divergence.

[0050] So far, the present invention has proposed a new criterion for checking the quality of reconstructed image texture. Unlike most previous criteria, the present method can use relative entropy to measure the divergence between image texture information in an interpretable way. The introduction of sliding windows can flexibly handle microscopic pixel displacements of textures, thus better approaching human perception.

[0051] In addition, based on the above evaluation of the texture quality of the reconstructed image, in an optional embodiment, in order to strengthen the model training, the present invention hopes to use TRE as a loss function, but according to the research of the present invention, this cannot be done directly. Unlike using TRE as an evaluation indicator, using TRE as one of the loss functions to supervise the training of deep autoencoders has more restrictions. Among them, the main difficulty is that the loss function itself must be differentiable. However, since the Canny edge detection algorithm used in the process of calculating TRE is not differentiable, TRE cannot be directly used as one of the loss functions.

[0052] To this end, the present invention provides a simple and effective alternative method, which uses another differentiable edge detection operator (such as the Sobel operator) to replace the Canny operator. In order to simplify the calculation and speed up the training process, the present invention omits the use of sliding windows during the training process, and uses the MSE loss to supervise the error between the results of applying the Sobel operator to the original image and the reconstructed image. This is because during the training process, the present invention hopes that the accuracy of the texture is as high as possible, so no pixel offset is accepted during the training process, and a loss function that focuses on pixel differences can be directly used. Based on the above, the new loss function used in this embodiment is: ; in, are two hyperparameters, is the algorithm used by the present invention to extract texture, namely the Sobel operator, represents the texture distribution of the source image, Represents the texture distribution of the reconstructed image. Experiments have shown that using such a loss function can significantly improve the texture quality.

[0053] Embodiment 2 See also Figure 5,Based on the above method, the present invention also provides a lightweight semantic communication system, the lightweight semantic communication system comprises: a sending end and a receiving end, the sending end and the receiving end establish a communication connection; The sending end is used to obtain a source image, extract texture semantics of the source image using a neural network, extract color semantics of the source image using a traditional visual algorithm, and send the color semantics and the texture semantics to the receiving end; The receiving end is used to fuse the color semantics and the texture semantics to reconstruct the source image.

[0054] Embodiment 3 This embodiment provides a storage medium, wherein the readable storage medium stores a lightweight image semantic communication program, and when the lightweight image semantic communication program is executed by a processor, the steps of the lightweight image semantic communication method described above are implemented.

[0055] In summary, the present invention adopts a simple traditional computer vision algorithm to replace the deep learning color semantic extraction model used in the semantic communication process of the prior art, thereby avoiding the significant high delay caused by the use of deep models, and solving the problem that the semantic communication method of the prior art completely adopts the neural network model, resulting in excessive computing power required and difficult to use under existing hardware conditions.

[0056] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0057] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, an optical disk, etc.

[0058] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A lightweight image semantic communication method, characterized in that: The lightweight image semantic communication method comprises: The sending end obtains a source image, uses a deep learning model to extract the texture semantics of the source image, and uses a traditional visual algorithm to extract the color semantics of the source image; The sending end sends the color semantics and the texture semantics to the receiving end; The receiving end fuses the color semantics and the texture semantics to reconstruct the source image.

2. A lightweight image semantic communication method according to claim 1, characterized in that: The sending end sends the color semantics and the texture semantics to the receiving end, specifically including: The sending end quantizes the color semantics into quantized color semantics and quantizes the texture semantics into quantized texture semantics; The sending end transmits the quantized color semantics and the quantized texture semantics to the receiving end; The receiving end uses a traditional visual algorithm to restore the quantized color semantics to obtain the color semantics, and uses a texture restoration model to process the quantized texture semantics to restore the texture semantics.

3. A lightweight image semantic communication method according to claim 2, characterized in that: The transmitting end and the receiving end both include a plurality of auxiliary devices, the deep learning model includes a texture extraction model and a shallow neural network, and the deep learning model is used to extract the texture semantics of the source image, specifically including: Inputting the source image into a shallow neural network to extract shallow features of the source image; Dividing the shallow features into a plurality of feature blocks; A number of the feature blocks are distributed to a plurality of auxiliary devices, and each auxiliary device processes the feature blocks respectively using a texture extraction model to obtain the texture semantics.

4. A lightweight image semantic communication method according to claim 3, characterized in that: The adopting a texture restoration model to process the quantized texture semantic restoration to obtain the texture semantics specifically includes: Semantically segmenting the quantized texture into a plurality of semantic blocks; The plurality of semantic blocks are distributed to a plurality of auxiliary devices, and each auxiliary device processes the semantic block respectively by using a texture restoration model to restore the texture semantics.

5. A lightweight image semantic communication method according to claim 4, characterized in that: The step of dividing the shallow features into a plurality of feature blocks specifically includes: Get the segmentation consumption model, and calculate the segmentation size of each auxiliary device based on the segmentation consumption model with the goal of minimizing the total time consumption: ； ； ； ； in, is the total time consumed, Indicates constraints, Indicates The memory consumption of each auxiliary device at the corresponding partition size, Indicates The maximum amount of memory available to auxiliary devices. is the maximum number of available devices, is the number of auxiliary devices; According to each segmentation size, the shallow feature is segmented into a plurality of feature blocks; The semantic segmentation of the quantized texture into a plurality of semantic blocks specifically includes: Get the segmentation consumption model, and calculate the segmentation size of each auxiliary device based on the segmentation consumption model with the goal of minimizing the total time consumption: ； ； ； ； According to each of the segmentation sizes, the quantized texture is semantically segmented into a plurality of semantic blocks.

6. A lightweight image semantic communication method according to claim 5, characterized in that: The obtaining of the segmentation consumption model specifically includes: Establish a parallel time-consuming model under the real protocol of homogeneous systems: ； in, The actual protocol parallel time of the homogeneous system, The number of CPU cycles required to complete the inference calculation for a unit of input data, is the transmission data rate between the receiving end and the auxiliary device, It is the data of auxiliary equipment rate, Indicates the number of CPU cycles that the auxiliary device can complete per unit time. is the size of the additional data to be transmitted. To quantify the semantic size of texture, The number of assistive devices available; Establish a parallel time-consuming model under the real protocol of heterogeneous systems: ； in, represents the actual protocol parallel time of the heterogeneous system, Representative auxiliary equipment The size of the data block to be processed, Represents the primary and secondary devices The transmission rate between Representative auxiliary equipment Data rate, Representative auxiliary equipment The number of CPU cycles that can be completed per unit time; Model memory consumption of auxiliary devices: ； in, The memory required to complete the inference calculation of the unit input data, The memory required to load the model parameter weights; The parallel time consumption model under the real protocol of the homogeneous system, the memory consumption model and the set of the parallel time consumption model under the real protocol of the heterogeneous system are used as the consumption model.

7. A lightweight image semantic communication method according to claim 3, characterized in that: The receiving end fuses the color semantics and the texture semantics to reconstruct the source image, specifically including: Inputting the color semantics and the texture semantics into a fusion model; The fusion model reconstructs and outputs the source image according to the color semantics and the texture semantics.

8. A lightweight image semantic communication method according to claim 7, characterized in that: The fusion model, the texture extraction model and the texture restoration model are obtained through training, and the loss function of the training is: ； in, represents the loss value, represents the source image, represents the reconstructed image, and To train the hyperparameters, represents the texture extraction operator, represents the texture distribution of the source image, Represents the texture distribution of the reconstructed image.

9. A lightweight semantic communication system, characterized in that: The lightweight semantic communication system comprises: a sending end and a receiving end, wherein the sending end and the receiving end establish a communication connection; The sending end is used to obtain a source image, extract texture semantics of the source image using a neural network, extract color semantics of the source image using a traditional visual algorithm, and send the color semantics and the texture semantics to the receiving end; The receiving end is used to fuse the color semantics and the texture semantics to reconstruct the source image.

10. A readable storage medium, characterized in that: The readable storage medium stores a lightweight image semantic communication program, and when the lightweight image semantic communication program is executed by a processor, the steps of the lightweight image semantic communication method according to any one of claims 1 to 8 are implemented.

Images