Image semantic communication method and device, equipment and storage medium

Abstract

The invention relates to the technical field of semantic communication, and provides an image semantic communication method, device and equipment and a storage medium, and the method is applied to a sending end, and comprises the steps: segmenting a to-be-sent target image into a plurality of image blocks; semantic information of each image block is extracted by using an encoder and input to a linear mapping layer for linear mapping processing, and then the semantic information is sent to a receiving end through a wireless channel; after a receiving end performs linear mapping on the semantic information, a decoder is used for performing semantic recovery and image reconstruction to obtain an output image; wherein the codec is provided with recursive variator units with the same structure, and the number of the recursive variator units is determined by performing joint training by taking calculation of time delay as an optimization target and taking similarity of an input initial sample image and a sample image reconstructed by a receiving end as a constraint. The network architecture of the codec is constructed through joint training, the terminal computing power and task performance can be balanced, and efficient semantic communication is realized under the condition that the terminal computing power is limited.

Description

Technical Field

[0001] The present invention relates to the field of semantic communication technology, and in particular to an image semantic communication method, device, equipment and storage medium. Background Art

[0002] With the integration and development of artificial intelligence and communication technology, semantic communication has become possible by using deep learning networks to extract semantic information from source data. As a new communication paradigm, semantic communication has attracted widespread attention due to its advantages such as its ability to transmit different semantic information for different needs and the bandwidth resources saved by transmitting only semantic information. It is expected to liberate communication networks from the traditional single solidified communication architecture based on data protocols and formats.

[0003] Semantic communication mainly relies on the extraction of semantic information by deep learning networks, and end-to-end communication is achieved through joint training of communication terminals. In terms of semantic extraction, images have more detailed and rich semantic information than texts, so they need to rely on a more complex network architecture. The complexity of the deep learning network structure is a key factor affecting the task performance and computing latency of end-to-end communication. Especially in application scenarios with strict latency requirements, deep learning networks need to accurately and efficiently process semantic information. Complex deep learning networks can effectively process semantic information, but the computing burden also increases. In particular, the computing power of communication terminals is different, and the computing power of terminals is closely related to computing latency. Deploying complex deep learning networks on terminals with poor computing power will greatly increase their computing latency. In current semantic communication, it is difficult to achieve a trade-off between the processing accuracy of semantic information and the complexity of the network, and it is impossible to balance the task performance and latency of semantic communication under the limited computing power of the terminal. Summary of the invention

[0004] The present invention provides an image semantic communication method, device, equipment and storage medium to solve the defects in the prior art that semantic communication relies on complex deep learning networks, it is difficult to achieve a trade-off between the processing accuracy of semantic information and the complexity of the network, and it is impossible to balance task performance and latency.

[0005] The present invention provides an image semantic communication method, which is applied to a sending end and comprises the following steps: Segment the target image to be sent to obtain multiple image blocks; Using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; The semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0006] According to the image semantic communication method provided by the present invention, the step of extracting features from each of the image blocks using an encoder to obtain semantic information of the target image includes: Using each recursive transformer unit in the encoder to extract features of the image blocks layer by layer to obtain semantic information of the target image; wherein any of the recursive transformer units includes a visual transformer layer, and the visual transformer layer includes a first normalization layer, a multi-head self-attention layer, a second normalization layer and a multi-layer perceptron layer; Based on a preset depth adjustment strategy, when extracting features from the image block at any target layer, an image stop mark of the image block at the target layer is generated based on a preset stop function; the target layer is any one of the visual changer layers; Determining whether it is necessary to continue to perform feature extraction processing on the image blocks according to the image stop mark; If it is necessary to continue to perform feature extraction processing on the image block, the image block output by the target layer is input into the subsequent layer of the target layer, and feature extraction processing is continued on the image block until it is determined according to the image stop mark that feature extraction processing does not need to be continued on the image block, and the semantic information of the image block is obtained.

[0007] According to the image semantic communication method provided by the present invention, the generating of the image stop mark of the image block at the target layer based on a preset stop function includes: Based on a preset stopping function, calculating a target stopping score of the image block at the target layer; Obtaining a cumulative stop score of the image block; the cumulative stop score is obtained by accumulating the stop scores of the image block in each visual changer layer before the target layer; Adding the target stop score to the cumulative stop score to obtain a target cumulative stop score of the image block at the target layer; Determine the target element corresponding to the image block in the preset marking vector. If the target cumulative stop score is greater than or equal to the preset stop threshold, set the target element to the first eigenvalue. If the target cumulative stop score is less than the preset stop threshold, set the target element to the second eigenvalue to obtain the image stop mark of the image block in the target layer.

[0008] According to the image semantic communication method provided by the present invention, the image blocks output by the target layer are input into the subsequent layer of the target layer, and feature extraction processing is continued on the image blocks until it is determined according to the image stop mark that feature extraction processing does not need to be continued on the image blocks, and semantic information of the image blocks is obtained, including: The image blocks output by the target layer are input into the subsequent layers of the target layer, and feature extraction processing is continued on the image blocks until it is determined according to the image stop mark that the image blocks do not need to be further subjected to feature extraction processing, and the stop scores and residual scores of the image blocks in each target visual changer layer are obtained; wherein the residual score of the Lth target visual changer layer is calculated based on the accumulated stop scores in the previous L-1 target visual changer layers, and the target visual changer layer is a visual changer layer that has subjected the image blocks to feature extraction processing; and L is a positive integer greater than 1; The weight value of the target visual changer layer is calculated according to the stop score and the remaining score of any of the target visual changer layers, and the semantic features of the image blocks output by each of the target visual changer layers are weighted and summed based on the weight value to obtain the semantic information of the image blocks.

[0009] According to the image semantic communication method provided by the present invention, before extracting features from each of the image blocks using an encoder to obtain semantic information of the target image, the method further includes: Acquire sample images, and divide each of the sample images into a plurality of sample image blocks; A sample data set is constructed based on the sample image blocks, and the encoder and the decoder are jointly trained using the sample data set to obtain an optimal combination of the number of recursive transformer units in the encoder and the decoder, as well as strategy parameters of a preset depth adjustment strategy.

[0010] According to the image semantic communication method provided by the present invention, the encoder and the decoder are jointly trained using the sample data set to obtain the optimal combination of the number of recursive transformer units in the encoder and the decoder, and the strategy parameters of the preset depth adjustment strategy, including: Initializing a first number of recursive transformer units in the encoder and a second number of recursive transformer units in a decoder of the receiving end; Based on a preset first number of iterations, the sample data set is input into an encoder for feature extraction to obtain sample semantic information of each of the sample images in the sample data set; The sample semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the sample semantic information into the second linear mapping layer for linear mapping processing, and then inputs it into the decoder for semantic restoration and image reconstruction to obtain a reconstructed sample image; Calculating the sample similarity between the reconstructed sample image and the sample image; Calculating a first computing delay of the transmitting end based on the computing capability of the transmitting end and the first number, and calculating a second computing delay of the receiving end based on the computing capability of the receiving end and the second number; If the sample similarity is greater than or equal to a preset similarity threshold, updating the first number and the second number according to the first calculation delay and the second calculation delay; determining an optimal combination of the number of recursive variator units in the encoder and the decoder according to the updated first number and second number; Based on the optimal combination, a preset depth adjustment strategy is added to the encoder and the decoder, and based on a preset second number of iterations, the encoder and the decoder are jointly trained using the sample data set to obtain strategy parameters of the depth adjustment strategy; the strategy parameters are used for the encoder and the decoder to calculate the stopping score of each visual transformer layer of each recursive transformer unit.

[0011] According to the image semantic communication method provided by the present invention, after segmenting the target image to be sent to obtain a plurality of image blocks, the method further comprises: Expanding each of the image blocks into an embedding vector; the vector dimension of the embedding vector is determined according to the size of the image block; The position information of the image blocks is embedded in the embedding vector; the position information is used to indicate the position of each of the image blocks in the target image.

[0012] The present invention also provides an image semantic communication device, comprising the following modules: An image segmentation module is used to segment the target image to be sent to obtain multiple image blocks; A feature extraction module, used to extract features from each of the image blocks using an encoder to obtain semantic information of the target image; A semantic sending module, used for inputting the semantic information into the first linear mapping layer for linear mapping processing, and then sending it to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic recovery and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the image semantic communication method as described above are implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-described image semantic communication methods are implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned image semantic communication methods are implemented.

[0016] The image semantic communication method, device, equipment and storage medium provided by the present invention divide the target image to be sent into multiple image blocks and input them into the encoder for feature extraction, and input the extracted semantic information into the first linear mapping layer for linear mapping processing, and then send it to the receiving end through a wireless channel. The receiving end inputs the received semantic information into the second linear mapping layer for mapping processing, and then uses the decoder to perform semantic recovery and image reconstruction to obtain the output image, thereby completing semantic communication. With the calculation delay as the optimization target and the similarity between the image sent by the sending end and the reconstructed image by the receiving end as the constraint, the number of recursive transformer units is determined by jointly training the encoder of the sending end and the decoder of the receiving end, and the network architecture of the encoder and decoder is constructed, the computing power of the terminal is taken into consideration, and a balance can be found between the computing power of the terminal and the task performance, ensuring efficient semantic transmission under the limited computing power of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the image semantic communication method provided by the present invention; Figure 2This is one of the flowcharts of the image semantic communication provided by the present invention; Figure 3 is a schematic structural diagram of a recursive transformer unit provided by the present invention; Figure 4 is a schematic diagram of the structure of the visual changer layer provided by the present invention; Figure 5 This is the second flow chart of the image semantic communication provided by the present invention; Figure 6 It is a structural schematic diagram of the image semantic communication device provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that from an epistemological point of view, information is divided into three levels, namely syntax, semantics and pragmatics. Shannon's classic communication model focuses on how the receiver can accurately restore the information sent by the source, that is, to achieve accurate transmission at the symbol level and ensure the integrity and accuracy of the information during transmission from the source to the receiver. Semantic communication is mainly task-oriented communication, which selectively extracts features, compresses and transmits the original signal. Compared with traditional communication, semantic communication only focuses on the part related to task performance, and focuses more on the semantic meaning of information, realizing understanding before transmission, which can greatly improve the transmission efficiency and reliability of communication.

[0021] End-to-end semantic communication is mainly based on deep learning technology to extract and recover semantic information. During the transmission process of semantic communication, the original data is extracted and encoded at the sender, and decoded and recovered by the receiver after transmission through the channel. The encoding and decoding process of semantic information mainly relies on neural networks to build an end-to-end semantic communication architecture from the sender to the receiver. By jointly training the semantic encoding and decoding of the sender and receiver, important task-related information in the signal can be learned, and adjustments can be made during the transmission process according to the wireless environment to improve communication efficiency and performance.

[0022] Furthermore, end-to-end semantic communication focuses on how to make full use of deep learning networks to extract semantic features from source data, especially the feature extraction and transmission of text, images, videos and other data for specific tasks. As far as image transmission is concerned, related technologies mainly aim to improve the performance of image transmission tasks, design end-to-end semantic encoding and decoding network architectures, and achieve effective extraction and recovery of semantic information. However, in actual communication processes, communication terminals are diverse, and image semantic communication also faces a series of challenges.

[0023] On the one hand, the depth of the deep learning network affects task performance and latency. The increase in network depth can usually improve the ability to fit semantic features, thereby enhancing task performance. However, the increase in network depth may also cause overfitting and significantly increase the amount of computation. A network depth that is too shallow may make it impossible to learn sufficiently deep features. Different terminals have different computing capabilities. As the network depth increases, the increase in computational complexity may cause a significant increase in computing latency. Therefore, end-to-end semantic communication requires the selection of a network of appropriate depth to achieve a trade-off between task performance and latency.

[0024] On the other hand, semantic communication is mostly based on static neural networks. After training, the network structure and parameters are fixed, and adaptive adjustments cannot be made for different inputs, which limits the generalization and adaptability of semantic communication. Specifically, in related technologies, the parameters and structure of deep learning networks are fixed after training. In the inference stage, static networks use the same network architecture and parameters for inference for all inputs. There are redundant calculations, resulting in low efficiency and poor generalization performance. Compared with static networks, dynamic networks refer to deep learning network architectures that can dynamically adjust their own structures or parameters when processing different inputs. Their dynamic structures include dynamic depth, dynamic width, and dynamic routing. Dynamic neural networks have many advantages, including selective activation of model components, significantly expanded parameter space, improved representation capabilities, a trade-off between accuracy and network complexity, high computational efficiency, and good generalization.

[0025] Semantic communication needs to comprehensively consider the network depth and the computing power of the communication terminal. It needs to find a balance between limited computing power and task performance to achieve more efficient semantic transmission and ensure efficient computing and good performance in different terminals and task scenarios. The dynamic network allows the semantic communication model to dynamically adjust the network structure during training and inference to perform semantic encoding and decoding of source data, which can more effectively extract and understand semantic information and further improve the efficiency and adaptability of image semantic communication.

[0026] Based on this, in order to solve the problems existing in the existing image semantic communication, the embodiment of the present invention proposes an image semantic communication method, which is based on a dynamic network to minimize the computing delay while ensuring the task performance, extract and restore semantic information based on the recursive transformer unit, consider the computing power of the transceiver, adaptively adjust the number of recursive transformer units, and design a stop mechanism to adaptively adjust the number of network layers used to process semantic information at both ends of the transceiver according to different input samples and channel conditions. The dynamic network can more effectively realize the extraction, transmission and understanding of semantic information, achieve a trade-off between accuracy and network complexity, improve computing efficiency and generalization ability, optimize the transmission of semantic information, and adapt to the dynamic changes of communication terminals and channels.

[0027] Specifically, Figure 1 is one of the flow charts of the image semantic communication method provided by the present invention, such as Figure 1 As shown, the image semantic communication method provided by the embodiment of the present invention is applied to the sending end in the semantic communication system, and the method specifically includes the following steps: Step 100, segmenting the target image to be sent to obtain multiple image blocks; Step 200, using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; Step 300, inputting the semantic information into the first linear mapping layer for linear mapping processing, and then sending it to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0028] First, the target image to be sent is segmented into a plurality of image blocks, and optionally, each image block has the same size. That is, the target image to be sent is evenly segmented into a plurality of image blocks according to a preset image size.

[0029] The encoder is used to extract features from each image block to obtain semantic information of the target image. Optionally, the encoder includes multiple recursive transformer units, which are recursive units constructed using a recursive method based on a layer of visual transformer layers. Each recursive transformer unit is constructed by recursively performing a layer of visual transformer layers once, and can realize the extraction of complex semantic information of the target image under the condition of limited storage capacity.

[0030] Furthermore, the semantic information is input into the linear mapping layer for linear mapping processing and then sent to the receiving end through the wireless channel, wherein after the semantic information is linearly mapped, the semantic information can be mapped into information transmittable by the wireless channel.

[0031] The receiving end recovers the received semantic information and reconstructs the image according to the recovered semantic information to obtain the output image and complete the communication.

[0032] In one embodiment, the transmitting end includes an encoder and a first linear mapping layer, the first linear mapping layer is used to perform linear mapping processing on the output of the encoder, and correspondingly, the receiving end includes a second linear mapping layer and a decoder, the second linear mapping layer is used to perform linear mapping processing on the information sent by the transmitting end through the wireless channel as the input of the decoder. Specifically, the receiving end inputs the received semantic information into the second linear mapping layer, uses the second linear mapping layer to perform linear mapping processing on the semantic information, and then uses the decoder to perform semantic recovery and image reconstruction on the semantic information after linear mapping processing to obtain an output image. The decoder corresponds to the transmitting end encoder. Optionally, the decoder of the receiving end also includes multiple recursive transformer units, and the recursive transformer units in the decoder have the same structure as the recursive transformer units in the encoder. The number of recursive transformer units in the decoder and the number of recursive transformer units in the encoder can be the same or different, and there is no specific limitation on this.

[0033] Optionally, the decoder at the receiving end has multiple recursive transformer units with the same structure as that in the encoder. In one embodiment, the number of recursive transformer units in the decoder is the same as the number of recursive transformer units in the encoder. In another embodiment, the number of recursive transformer units in the decoder is different from the number of recursive transformer units in the encoder. In some embodiments, the number of recursive transformer units in the decoder is the same as the number of recursive transformer units in the encoder, and the structure of the multiple recursive transformer units in the encoder is the same as that in the decoder, that is, the decoder and the encoder have the same structure, which is based on multiple recursive transformer units.

[0034] Optionally, the number of recursive transformer units in the encoder and the decoder is determined by jointly training the encoder at the transmitting end and the decoder at the receiving end, taking the computational delay as an optimization target and the similarity between the initial sample image sent by the transmitting end and the output sample image reconstructed by the receiving end as a constraint. Optionally, the number of recursive transformer units in the decoder can be determined based on the number of recursive transformer units in the encoder, or can be determined by joint training of the transmitting end and the receiving end when determining the number of recursive transformer units in the encoder.

[0035] In one embodiment, the transmitter and the receiver determine the number of recursive transformer units in the encoder and the number of recursive transformer units in the decoder through joint training. During the joint training, the number of recursive transformer units is optimized with the computational delay as the optimization target and the similarity between the initial sample image input by the transmitter and the reconstructed sample image output by the receiver as the constraint to obtain the number of recursive transformer units in the encoder and the number of recursive transformer units in the decoder. It can be seen that the computational delay is related to the computing power of the transmitter and the receiver.

[0036] In this embodiment, the target image to be sent is divided into multiple image blocks and input into the encoder for feature extraction, and the extracted semantic information is input into the linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel. After the receiving end performs linear mapping processing on the received semantic information, the decoder is used to perform semantic recovery and image reconstruction to obtain an output image, thereby completing semantic communication. By taking the computational delay as the optimization target and the similarity between the image sent by the sending end and the reconstructed image by the receiving end as the constraint, the encoder at the sending end and the decoder at the receiving end are jointly trained, the number of recursive transformer units in the codec is determined, and the network architecture of the encoder and decoder is constructed, the processing accuracy and network complexity are weighed, and the computing power of the terminal is considered. A balance can be found between the computing power of the terminal and the task performance of semantic communication, ensuring efficient semantic communication under the condition of limited computing power of the terminal.

[0037] In one embodiment, referring to Figure 2 In the image semantic communication process shown, the sending end includes an image block embedding layer, a position information embedding layer, an L-layer recursive transformer unit and a first linear mapping layer. When performing image semantic communication, the sending end inputs the input image into the encoder for encoding processing. Specifically, the input image is first divided into multiple image blocks by the image block embedding layer, and the position information is embedded in the image blocks by using the position information embedding layer. The position information embedding layer can be constructed by using a learnable one-dimensional position encoder annotation or the like. The embedded position information is used to characterize the position of each image block in the input image; then, the image blocks are feature extracted by the L-layer recursive transformer unit to obtain the semantic information of the input image. Finally, the extracted semantic information is encoded into data transmittable by the wireless channel through the processing of the first linear mapping layer and sent to the receiving end. Among them, the input image is the target image to be sent.

[0038] The receiving end inputs the received data into the decoder for decoding processing. Specifically, the receiving end includes a second linear mapping layer, an L-layer recursive transformer unit and an image reconstruction layer receiving end. During the decoding processing, the receiving end first decodes the received data through the second linear mapping layer to restore the semantic information; then, the semantic information is decoded layer by layer through the L-layer recursive transformer unit in the decoder, and finally the decoded semantic information is reconstructed into an image using the image reconstruction layer to obtain the output image of the receiving end.

[0039] Based on this, in step 100, after dividing the target image to be sent into a plurality of image blocks, the following steps may also be included: Step 101, expanding each of the image blocks into an embedded vector; the vector dimension of the embedded vector is determined according to the size of the image block; Step 102: embedding the position information of the image blocks in the embedding vector; the position information is used to indicate the position of each image block in the target image.

[0040] After the target image is divided into multiple image blocks, each image block is expanded into an embedding vector, the vector dimension of the embedding vector is determined according to the image size of the image block, and the position information of the image block is embedded in the embedding vector, and the position information represents the position of the image block in the target image to be sent.

[0041] In one embodiment, the target image to be sent The image size is ,in, is the image height, is the image width, is the number of channels of the target image. The input target image is divided into blocks of the same size through the image block embedding layer. Multiple image blocks, each image block includes channels. The number of image blocks after segmentation is Each image block is flattened into a one-dimensional embedding vector with a dimension of .definition The vector dimension corresponding to each image block is , after the image block embedding layer, the image can be represented as .

[0042] Then, the position information embedding layer is used to embed the position information of each block, that is, the position information of the image block in the target image is embedded in the embedding vector using the position information embedding layer to retain the spatial information in the original target image.

[0043] Furthermore, the semantic encoder is composed of a series of recursive transformer units. The embedding vectors of the image blocks are processed layer by layer by the encoding layer in the recursive transformer unit to extract their semantic information. The self-attention mechanism of the recursive transformer unit is used to strengthen the information exchange between different image regions. The processed semantic information is passed through the linear mapping layer to map the high-dimensional semantic information into symbols suitable for wireless channel transmission. These symbols are sent to the receiving end through the wireless channel. The corresponding vectors of each image block are mapped to the corresponding vectors of each image block. The dimensional embedding vector is mapped to dimensional transmission vector, the bandwidth compression ratio (Channel Bandwidth Ratio, CBR) can be expressed as: ; (1)

[0044] CBR is used to describe the compression degree of the original features after encoding relative to the original source in joint source channel coding. The receiver first linearly maps the semantic information after channel transmission, recovers the received semantic information, and then decodes the semantic information layer by layer through a sequence of recursive transformer units similar to the encoder. Finally, the decoded semantic information is reconstructed into an image using the image reconstruction layer to obtain the restored output image. .

[0045] In order to minimize the computational delay in semantic communication, the optimal number of recursive transformer units is selected according to the computing power of the sender and receiver during the training phase, which can initially reduce the computational delay by reducing the model complexity. However, since each recursive transformer unit contains the recursion of the visual transformer layer, the possibility of further reducing the computational delay on a single recursive transformer unit is limited. To further reduce the computational delay, a stopping mechanism is used to enable the network to dynamically determine the number of visual transformer layers required for feature extraction for each input image, allowing early stopping during the feature extraction process, thereby controlling the network depth at both ends of the codec to adapt to different input images and channel conditions, avoiding redundant calculations to improve computational efficiency.

[0046] Specifically, in step 200, the encoder is used to extract features from each image block to obtain semantic information of the target image, including: Step 210, using each recursive transformer unit in the encoder to extract features of the image blocks layer by layer to obtain semantic information of the target image; wherein any of the recursive transformer units includes a visual transformer layer, and the visual transformer layer includes a first normalization layer, a multi-head self-attention layer, a second normalization layer and a multi-layer perceptron layer; Step 220, based on a preset depth adjustment strategy, when performing feature extraction on the image block at any target layer, generating an image stop mark of the image block at the target layer based on a preset stop function; the target layer is any one of the visual changer layers; Step 230, determining whether it is necessary to continue feature extraction processing on the image blocks according to the image stop mark; Step 240: If it is necessary to continue to perform feature extraction processing on the image block, the image block output by the target layer is input into the subsequent layer of the target layer, and feature extraction processing is continued on the image block until it is determined according to the image stop mark that feature extraction processing is no longer necessary, thereby obtaining semantic information of the image block.

[0047] The recursive transformer units in the encoder are used to extract features from the image blocks layer by layer to obtain the semantic information of the target image. Figure 3 , the recursive mutator unit and the visual mutator layer have a one-to-one correspondence, and a recursive mutator unit includes a visual mutator layer, and a recursive method is used to recursively perform a layer of visual mutator layers to form a recursive mutator unit. Optionally, a recursive mutator unit has a correspondence with multiple visual mutator layers, that is, a recursive mutator unit includes multiple visual mutator layers, and a recursive method is used to recursively perform multiple visual mutator layers to form a recursive mutator unit.

[0048] Furthermore, the visual transformer layer includes a first normalization layer, a multi-head attention layer, a second normalization layer and a multi-layer perceptron layer. The network structure is as follows: Figure 4 As shown. When any target layer in each visual transformer layer corresponding to the recursive transformer unit performs feature extraction on an image block, an image stop mark of the image block at the target layer is generated based on a preset stop function, and whether the image block needs to continue feature extraction processing is determined according to the image stop mark. The image stop mark of the image block is used to indicate whether the image block needs to continue feature extraction, and the image stop mark can be generated according to the degree of information extraction of the image block.

[0049] If it is necessary to continue to perform feature extraction processing on the image blocks, the image blocks output by the target layer are input into the subsequent layer of the target layer, and feature extraction processing is continued on the image blocks until it is determined according to the image stop mark that feature extraction processing is no longer needed on the image blocks, and the semantic information of the image blocks is obtained. After obtaining the semantic information of each image block, the semantic information of the target image is acquired.

[0050] Further, in step 220, based on a preset stop function, an image stop mark of the image block in the target layer is generated, specifically including: Step 221, calculating a target stopping score of the image block at the target layer based on a preset stopping function; Step 222, obtaining a cumulative stop score of the image block; the cumulative stop score is obtained by accumulating the stop scores of the image block in each visual changer layer before the target layer; Step 223, adding the target stop score to the cumulative stop score to obtain the target cumulative stop score of the image block at the target layer; Step 224, determine the target element corresponding to the image block in the preset marking vector. If the target cumulative stop score is greater than or equal to the preset stop threshold, set the target element to the first eigenvalue. If the target cumulative stop score is less than the preset stop threshold, set the target element to the second eigenvalue to obtain the image stop mark of the image block in the target layer.

[0051] First, based on a preset stop function, the target stop score of the image block in the target layer is calculated to obtain the cumulative stop score of the image block, wherein the cumulative stop score of the image block is obtained by accumulating the stop scores of the image block in each visual changer layer before the target layer.

[0052] Furthermore, the target stop score of the image block in the current target layer is added to the cumulative stop score to obtain the target cumulative stop score of the image block in the current target layer, and an image stop mark of the image block in the current target layer is generated according to the target cumulative stop score. The method of generating the image stop mark includes but is not limited to resetting the element value of the mark vector. Specifically, first, the target element corresponding to the image block in the preset mark vector is determined. In the mark vector, each element corresponds to an image block, and the mark vector corresponds to each image block of the input target image. The element value of the target element is set according to the calculated target cumulative stop score. Optionally, if the target cumulative stop score is greater than or equal to the preset stop threshold, the target element is set to the first eigenvalue. If the target cumulative stop score is less than the preset stop threshold, the target element value is set to the second eigenvalue, thereby obtaining the image stop mark of the image block in the current target layer.

[0053] The first eigenvalue is used to indicate that it is not necessary to use the subsequent layers of the target layer to perform feature extraction processing on the image block, and the second eigenvalue is used to indicate that it is necessary to use the subsequent layers of the target layer to continue to perform feature extraction processing on the image block. Exemplarily, the first eigenvalue is 1 and the second eigenvalue is 0.

[0054] In some embodiments, the semantic information of the image block is obtained by weighted summing the outputs of each visual changer layer, and the weighted weight value is determined according to the stop score and the residual score calculated by the visual changer layer during feature extraction. In step 224, when it is determined that the image block needs to continue to be subjected to feature extraction processing, the image block output by the target layer is input into the subsequent layer of the target layer, and the image block continues to be subjected to feature extraction processing until it is determined according to the image stop mark that the image block does not need to be subjected to feature extraction processing, and the semantic information of the image block is obtained, which may also include: Step 2241, input the image block output by the target layer into the subsequent layer of the target layer, continue to perform feature extraction processing on the image block, until it is determined according to the image stop mark that the image block does not need to be further subjected to feature extraction processing, and obtain the stop score and residual score of the image block in each target visual changer layer; wherein the residual score of the Lth target visual changer layer is calculated based on the cumulative stop score in the previous L-1 target visual changer layers, and the target visual changer layer is a visual changer layer that performs feature extraction processing on the image block; L is a positive integer greater than 1; Step 2242, calculate the weight value of the target visual changer layer according to the stop score and the remaining score of any of the target visual changer layers, and perform weighted summation on the semantic features of the image blocks output by each of the target visual changer layers based on the weight value to obtain the semantic information of the image blocks.

[0055] The image blocks output by the target layer are input into the subsequent layers of the target layer, and feature extraction processing is continued on the image blocks. In the feature extraction processing on the image blocks in the subsequent layers, the stop scores of the image blocks are calculated at each layer, as in the target layer, and then the image stop marks of the image blocks are generated to determine whether to continue feature extraction processing on the image blocks. Until it is determined that the image blocks do not need to continue feature extraction processing, the stop scores and residual scores of the image blocks in each target visual changer layer are obtained, wherein the residual score of the Lth target visual changer layer is calculated based on the cumulative stop scores in the first L-1 target visual changer layers, and L is a positive integer greater than 1. The target visual changer layer is the visual changer layer that performs feature extraction processing on the image blocks among the visual changer layers.

[0056] Furthermore, according to the stopping score and the remaining score of any target visual transformer layer, the weight value of the target visual transformer layer is calculated, and based on the weight value, the semantic features of the image blocks output by each target visual transformer layer are weighted and summed to obtain the semantic information of the image blocks.

[0057] It should be noted that when the decoder at the receiving end restores semantic information, it also calculates the stop score of the image block in each visual transformer layer based on the preset stop function, and generates an image stop mark for the image block according to the stop score, thereby determining whether it is necessary to continue semantic restoration of the image block based on the image stop mark. It adopts the same depth adjustment strategy as the encoder end, which will not be repeated here.

[0058] In one embodiment, the image blocks entering the recursive transformer unit in the codec are represented as ,in is the number of image blocks, is the embedding vector dimension. If the encoder or decoder has Recursive transformer units, based on the recursion of the visual transformer layer, the encoder or decoder contains a total of The image blocks are processed layer by layer through the visual transformer layer. The input of the layer visual transformer is the output of the previous layer. , after the The semantic features output after layer calculation can be expressed as: ; (2)

[0059] in, Representative Layer Visual Mutator Layer, Represents image blocks After The output after the layer visual transformer layer.

[0060] Considering the differences in content and features between image blocks, different image blocks may need to go through different calculation layers. Therefore, during the calculation process, it is necessary to mark which image blocks need to continue to participate in the calculation and which image blocks stop calculating at the current layer. Based on this, a marking vector is set for the input target image. Image stop marker for generating image blocks, indicating that after the Whether to continue calculating image blocks after the layer network , illustratively, Indicates to continue the next layer of calculation. Indicates to stop the next layer of calculation. For the convenience of expression, another marker vector is defined , and On the contrary, . And, define a Dimensional vector For statistical The image blocks are taken from the first recursive transformer unit. Layer Visual Transformer Layer to The number of times the layer participates in the calculation, marked Represents image blocks Participate in Layer calculation, It can be expressed as: ; (3)

[0061] In order to adaptively adjust the network depth of the visual transformer layer in the recursive transformer unit, a stop function is set. The stop function calculates the stop score of each image block according to the output of each visual transformer layer, which is used to indicate the probability of stopping further calculation of each image block after each layer of calculation. Since the transmission signal passes through the channel, the channel condition affects the degree of distortion of the transmission signal, that is, the transmission signal distortion is smaller when the channel condition is better, and the transmission signal distortion is larger when the channel condition is worse. In semantic communication, it is possible to consider increasing the number of network layers. The multi-layer network structure helps to extract and reconstruct semantic features more finely to improve the accuracy and completeness of information transmission in a noisy environment. The stop score of each image block can be expressed as: ; (4)

[0062] in, Representative The image is divided into blocks through The stopping score after layer calculation, Representative The image is divided into blocks through The output of the layer. Use a fully connected layer to linearly transform the output, weight and the bias term Yes The parameters for the linear transformation are shared among different layers of the visual transformer network. is based on the channel noise variance The calculated weight parameters are: is a hyperparameter, through To adjust the output of the stop function to adapt to changes in channel conditions. It can be obtained through channel estimation and used as a known condition input during the training process.

[0063] After the above calculations, activation functions such as Sigmoid are used to ensure that the stop score is limited to the range of 0 to 1, indicating the probability of stopping the calculation of each image block at the current layer. Calculating the stop score for each image block takes into account the potential differences between different image blocks, which can more accurately reflect the information retention requirements of the image block during its transmission process.

[0064] In order to determine the image block After After the layer calculation, is it necessary to perform the next layer of network calculation? Layer to Cumulative stop score for the layer visual changer layer ,Right now: ; (5)

[0065] When the accumulation stops When it is larger, it indicates that the image is blocked. After If further processing after layer network calculation cannot significantly improve the output feature quality, in this case, the calculation of the image block in subsequent layers can be stopped to improve the calculation efficiency. Based on this, a stopping threshold is set. , as a benchmark for determining whether to continue the calculation, used to generate the image stop mark for the image block. ,according to and Determine whether to participate in the next layer of calculation. Exceeding the setting When , the subsequent calculation of the image block will be stopped, otherwise, the next layer of calculation will be continued. The image block is generated and updated by the following formula 6 n Image tags: ; (6)

[0066] because and On the contrary, according to the above formula, the image stop mark of the image block can be obtained ,Right now: ; (7)

[0067] The image stop marks of multiple image blocks of the input image constitute a mark vector, and the image stop mark of each image block That is, the element value of the corresponding element of the image block in the label vector. Further, the remainder of the stop score is defined as the residual score, which is used to reflect the amount of useful information that the image block can still provide after a certain number of layers of calculation, and helps to determine whether the image block needs to be further passed to the deeper network of the visual transformer layer. The calculation method of the residual score is: ; (8)

[0068] The final output of the network of recursive transformer units is obtained by weighted summing the outputs of each visual transformer layer. The weight value is calculated by the stop score and the residual score of each layer, which represents the relative importance of feature extraction at each layer. Layer Visual Transformer layer output weights It can be expressed as: ; (9)

[0069] in, Represents element-by-element multiplication, through and Distinguish the image blocks that need to continue to be calculated and the image blocks that stop calculation. The stop score representing each image block can be calculated by formula 4. Represents the residual score of each image block and can be calculated by Formula 8.

[0070] During the back-propagation process, the gradient is adjusted according to the weights of the outputs of each layer. After calculating the output weights of each layer, the output of each visual transformer layer can be expressed as: . (10)

[0071] In some embodiments, the dynamic adjustment process of the network depth of the visual transformer layer is as shown in the following algorithm: Input: 3D tensor Stop Threshold ; Output: 3D tensor , calculate the number of times , the remaining score ; 1. Initialization: Image block marking and , cumulative stop score , the number of recursive transformer units L ; for do: 2. Update the calculation times by image tagging ; Calculate the Output of the layer visual transformer layer ; 3. According to The layer outputs the calculation stop score when When , the stop score is calculated according to Formula 4 ,otherwise ; 4. Calculate the residual score of the image block ; 5. Add the stop scores and get the cumulative stop score ; 6. Update image mark based on cumulative stop score When When , it is marked as 0; 7. Update image stop mark : When marked as 1, When marked as 0; 8. Calculate the output weight ; 9. Update output ; 10. Return to 2 until the image stops marking Mark it as 1.

[0072] In the above algorithm, since the recursive transformer unit recurses the visual transformer layer once, the L recursive transformer units include L visual transformer layers. Based on the recursion of the visual transformer layer, it is equivalent to performing 2L layers of processing on the image blocks, that is, it is equivalent to having 2L visual transformer layers.

[0073] Based on this, during the encoding process of the encoder, a feature extraction operation of encoding each image block layer by layer is performed, and the image stop mark is calculated based on the preset depth adjustment strategy to determine the processing depth of the image block. After the processing is completed, the stop score and the residual score of each visual changer layer are obtained, the corresponding weight value is calculated, and the output of each visual changer layer is weighted and summed to obtain the extracted semantic information. The extracted semantic information is mapped into symbols that can be transmitted by the wireless channel through the linear mapping layer and sent to the receiving end. The receiving end also uses the linear mapping layer to linearly map the received symbols to obtain the encoded semantic information, and then adopts the same depth adjustment strategy as the encoder to perform a feature recovery operation of decoding the semantic information layer by layer in the decoder to restore the semantic information of the image block. Finally, after all the image blocks are decoded, the decoded semantic information is reconstructed into an image using the image reconstruction layer to obtain the output image of the receiving end. The processing depth when decoding the image block can be the same as the processing depth when encoding, and is not specifically limited here.

[0074] In one embodiment, Figure 5 is another flow chart of image semantic communication provided by an embodiment of the present invention, such as Figure 5 As shown, based on the variance estimation of the wireless channel, the noise variance of the wireless channel between the transmitter and the receiver is determined, and the noise variance is used for the stopping function to calculate the stopping scores of each layer of the visual transformer layer in the encoding and decoding stage, thereby determining the network depth of the visual transformer layer required when processing the image blocks in the encoding and decoding stage.

[0075] In one embodiment, before performing semantic communication, the encoder at the transmitting end and the decoder at the receiving end need to be jointly trained to determine the number of recursive transformer units in the encoder and the decoder, as well as the policy parameters of the preset depth adjustment policy. Before step 200, it also includes: Step 201, obtaining sample images, and dividing each of the sample images into a plurality of sample image blocks; Step 202, constructing a sample data set based on the sample image blocks, and using the sample data set to jointly train the encoder and the decoder to obtain the optimal combination of the number of recursive transformer units in the encoder and the decoder, as well as the strategy parameters of the depth adjustment strategy.

[0076] First, a sample image is obtained, each sample image is divided into a plurality of sample image blocks, a sample data set is constructed based on the sample image blocks, the sample data set is input into an encoder, and the encoder at the transmitting end and the decoder at the receiving end are jointly trained using the sample data set, so as to obtain an optimal combination of the number of recursive transformer units in the encoder and the decoder, and a policy parameter of the depth adjustment strategy. Optionally, the optimal combination includes a first optimal number of recursive transformer units in the encoder at the transmitting end, and a second optimal number of recursive transformer units in the decoder at the receiving end. The first optimal number and the second optimal number may be the same or different.

[0077] Optionally, when dividing the sample image into sample image blocks, the specific processing method can be the same as that of the target image, that is, the image block embedding layer in the encoder is used to segment the sample image, and each sample image is segmented into multiple sample image blocks. In addition, after segmenting the sample image blocks, the sample image blocks can be further expanded into corresponding sample embedding vectors through the position information embedding layer of the encoder, and the position information of the sample image blocks is embedded in the sample embedding vector. The position information is used to characterize the position of the sample image block in the sample image.

[0078] In some embodiments, in step 202, the encoder at the transmitting end and the decoder at the receiving end are jointly trained using the sample data set to obtain the optimal combination of the number of recursive transformer units in the encoder and the decoder, and the strategy parameters of the preset depth adjustment strategy, specifically including: Step 2021, initializing a first number of recursive variator units in the encoder and a second number of recursive variator units in the decoder of the receiving end; Step 2022: Based on a preset first number of iterations, input the sample data set into an encoder for feature extraction to obtain sample semantic information of each sample image in the sample data set; Step 2023, inputting the sample semantic information into the first linear mapping layer for linear mapping processing, and then sending it to the receiving end through a wireless channel; the receiving end inputs the sample semantic information into the second linear mapping layer for linear mapping processing, and then inputs it into the decoder for semantic restoration and image reconstruction, to obtain a reconstructed sample image; Step 2024, calculating the sample similarity between the reconstructed sample image and the sample image; Step 2025: Calculate a first computing delay of the transmitting end based on the computing capability of the transmitting end and the first number, and calculate a second computing delay of the receiving end based on the computing capability of the receiving end and the second number; Step 2026: If the sample similarity is greater than or equal to a preset similarity threshold, update the first number and the second number according to the first calculation delay and the second calculation delay; Step 2027, determining an optimal combination of the number of recursive transformer units in the encoder and the decoder according to the updated first number and the second number; Step 2028: Based on the optimal combination, a preset depth adjustment strategy is added to the encoder and the decoder, and based on a preset second number of iterations, the encoder and the decoder are jointly trained using the sample data set to obtain strategy parameters of the depth adjustment strategy; the strategy parameters are used for the encoder and the decoder to calculate the stopping score of each visual transformer layer of each recursive transformer unit.

[0079] When jointly training the encoder and the decoder, first initialize the first number of recursive transformer units in the encoder and the second number of recursive transformer units in the decoder at the receiving end, and input the constructed sample data set into the encoder for feature extraction based on a preset number of iterations to obtain sample semantic information of each sample image in the sample data set. Input the extracted sample semantic information into the linear mapping layer for linear mapping processing, and then send it to the receiving end through a wireless channel. After the receiving end performs linear mapping processing on the sample semantic information, it is input into the decoder, and semantic restoration is performed by the decoder, and image reconstruction is performed based on the restored sample semantic information to obtain a reconstructed sample image.

[0080] Furthermore, the sample similarity between the reconstructed sample image and the input sample image is calculated, the first computing delay of the transmitting end is calculated based on the computing power of the transmitting end and the first number of recursive transformer units in the transmitting end encoder, and the second computing delay of the receiving end is calculated based on the computing power of the receiving end and the second number of recursive transformer units in the receiving end decoder.

[0081] Taking sample similarity as a constraint and minimizing the computational delay of the transmitting end and the receiving end as an optimization goal, the number of recursive transformer units in the encoder and the number of recursive transformer units in the decoder are optimized and solved to obtain the best combination of the number of recursive transformer units in the encoder and the decoder. The best combination includes a first optimal number of recursive transformer units in the encoder solved according to the computing power of the transmitting end, and a second optimal number of recursive transformer units in the decoder solved according to the computing power of the receiving end.

[0082] Optionally, when determining the optimal combination of the number of recursive transformer units, sample similarity is used as a constraint and minimization of computational delay is used as an optimization goal. Specifically, if the sample similarity is greater than or equal to a preset similarity threshold, the first number and the second number of recursive transformer units are updated according to the first computational delay of the transmitting end and the second computational delay of the receiving end. Optionally, the first number of recursive transformer units is updated according to the first computational delay of the transmitting end, and the second number of recursive transformer units is updated according to the second computational delay of the receiving end; optionally, the total delay is calculated according to the first computational delay of the transmitting end and the second computational delay of the receiving end, and the first number and the second number of recursive transformer units are updated according to the total delay. After completing the preset number of iterations, the optimal combination of the number of recursive transformer units in the encoder and the decoder is determined according to the last updated first number and second number, and the optimal combination includes the first number and the second number after the final update.

[0083] In one embodiment, the joint training of the encoder and decoder includes two stages. In the first stage, the best combination of the number of recursive transformer units in the encoder and decoder is determined through joint training. In the second stage, after the number of recursive transformer units in the encoder and decoder is determined, a deep adjustment strategy is added to fine-tune the network parameters of the encoder and decoder.

[0084] It should be noted that after the depth adjustment strategy is added in the second stage of joint training, the feature extraction process of the encoder for the sample image is the same as that for the target image, and the decoding process of the decoder for the sample image is also the same as that for the target image. Both calculate the stop score based on the stop function and further generate an image stop mark to control the network depth of the recursive transformer unit, which will not be repeated here.

[0085] In some embodiments, the goal of semantic communication is related to a specific task. In various embodiments of the present invention, the image reconstruction task is mainly considered, and the network parameters of each part of the encoder at the transmitting end and the decoder at the receiving end are jointly trained using MSE (mean-square error) as a loss function. The loss function is specifically expressed as shown in the following formula 11: ; (11)

[0086] The number of recursive transformer units in the codecs at both the transmitter and receiver can be set to a fixed value. In semantic communication, the number of recursive transformer units affects task performance and computational latency. The increase in the number of recursive transformer units can extract deeper semantic information, but it will also increase computational latency. To balance task performance and latency requirements, the optimal number of recursive transformer units must be selected to ensure task performance while minimizing computational latency. During joint training, the similarity between the reconstructed image and the sample image is evaluated by the Structure Similarity Index Measure (SSIM). To ensure that the task performance SSIM is greater than the set threshold, the optimization objective can be expressed as: ; (12)

[0087] st ; (13)

[0088] in, and They represent the computational delays at the sender and receiver, respectively, and are related to the number of recursive transformer units at the sender and receiver and the computing power of the terminal. denote the number of recursive transformer units at the sending and receiving ends, respectively, Represents the structural similarity index value between the image restored by semantic communication transmission and the input image, is the similarity threshold set to ensure image quality.

[0089] Since both image block embedding and position information embedding are linear calculations, their computational complexity is less than that of multiple recursive transformer units. Therefore, considering the computational complexity and computational latency, the computational complexity of the recursive transformer unit is the main one. The computational complexity of a visual transformer layer is expressed as , the amount of calculation is related to the size of the input image, the dimension of the visual features, and the number of attention heads. For the convenience of analysis, without considering the memory access time, the calculation delay is only related to the amount of calculation and the computing power of the terminal. The time required for a visual transformer layer calculation can theoretically be expressed as: ; (14)

[0090] in, represents the computing power of the terminal. Assuming that the computing power of the sending end and the receiving end is different, the number of recursive transformer units at both ends is and , and the computing power is expressed as and , then according to formula (14), the calculation time at both ends can be calculated as: ; (15)

[0091] ; (16)

[0092] Through the traversal method in the joint training process, the optimal combination that satisfies the similarity constraint can be found The specific traversal process is as follows: When starting the traversal, set and The initial value of is 1 and is updated after each iteration. and The value of Combine, calculate the computational delay at both ends according to formula 15-16, and evaluate through joint training Combine the image restoration performance during semantic communication and determine whether the SSIM is greater than the set similarity threshold. If the condition is met, compare the calculated delay between the sender and the receiver with the minimum delay. After possible combinations, choose the one with the lowest latency The combination is the optimal combination, where the minimum delay is the minimum value of the calculation delay selected from the completed iteration process during the joint training process.

[0093] The specific implementation process is shown in the following algorithm: enter: ; Output: Optimal combination , ; 1. Initialization: ; 2. for ; 3. for ; 4. Get the current combination through joint training Image restoration performance under ; 5. if , the calculation delay between the sender and the receiver is calculated by formula 15-16 and ; if ; 6. Update the optimal number of recursive transformer units on the sender ; 7. Update the optimal number of recursive transformer units at the receiving end ; 8. Update the minimum calculation delay ; Return to 4 until the Traverse the combination to get the optimal combination And its corresponding minimum computation delay .

[0094] In some embodiments, the joint training process of the transmitter and the receiver mainly includes two stages. The first stage is to perform end-to-end image semantic communication training. As the loss function, the constructed sample data set is used as input to train the number of recursive transformer units at both ends of the codec to extract semantic information from the image and achieve image reconstruction. Through end-to-end joint training, the network parameters of the encoder and decoder are optimized to achieve the best communication performance. In the second stage, a deep adjustment strategy based on the stopping function is added to calculate the stopping probability of the output results of each layer of the visual transformer layer. The final feature extraction result is obtained according to the weighted sum of the outputs of each layer, and the calculated loss is added to the original loss function for further training. Among them, the second stage mainly trains the parameters in Formula 4, and fine-tunes the network parameters of the first stage.

[0095] In the second stage of joint training, based on the obtained optimal combination of the number of recursive transformer units in the codec, a depth adjustment strategy is added to the codec. The depth adjustment strategy is constructed based on the stopping function and has initial strategy parameters. The initial strategy parameters include the first weight of each visual transformer layer of each of the recursive transformer units in the encoder and the second weight of each visual transformer layer of each of the recursive transformer units in the decoder.

[0096] Optionally, the policy parameters include a linear transformation parameter of a stop function of a preset depth adjustment policy, and the linear transformation parameter is used for the codec to calculate the stop score of each visual transformer layer. Further, the linear transformation parameter includes a first weight and a second weight, and the first weight and the second weight are the linear transformation weights of the stop function in the depth adjustment policy, such as the weights shown in Formula 4 , wherein the first weight is the linear transformation weight of the visual transformer layer of each recursive transformer unit in the encoder, which is used for the visual transformer layer of each recursive transformer unit in the encoder to calculate the stopping score, and the second weight is the linear transformation weight of the visual transformer layer of each recursive transformer unit in the decoder, which is used for the visual transformer layer of each recursive transformer unit in the decoder to calculate the stopping score.

[0097] Furthermore, based on the preset second number of iterations, the codec is jointly trained using a sample data set. In each round of joint training, according to the preset depth adjustment strategy, the first stopping probability of each visual transformer layer of each recursive transformer unit in the encoder and the second stopping probability of each visual transformer layer of each recursive transformer unit in the decoder are calculated, and the first stopping probability and the second stopping probability are both determined according to the calculated stopping score. The feature extraction result of the codec is obtained by weighted summing the outputs of each visual transformer layer, and the loss value of this round of iteration is obtained. Based on the loss value, the policy parameters of the depth adjustment strategy are adjusted, that is, the weights of each visual transformer layer of the codec are adjusted. When the number of iterations reaches the preset second number of iterations, or the loss value meets the preset accuracy requirements, the second stage of joint training of the codec is completed to obtain the policy parameters of the depth adjustment strategy.

[0098] In this embodiment, through joint training, the number combinations of recursive transformer units at the sending end and the receiving end are traversed, and the optimal combination with the smallest computing delay is selected. This can optimize the allocation of computing resources and minimize the initial computing delay in units of recursive transformer units. This is beneficial to achieving optimal task performance while minimizing computing delay under the condition of limited terminal computing power, thereby ensuring efficient semantic communication.

[0099] Furthermore, in the process of semantic communication, the stop score is calculated at each visual transformer layer as the stop probability of the image block for the next layer of calculation, and a corresponding stop mark is generated to indicate whether to continue calculating the image block. A dynamic stop mechanism is adopted to calculate the stop probability for the output of each layer of the network, so that the semantic extraction network in the encoder and the semantic recovery network in the decoder can dynamically adjust the network depth according to the input image characteristics and channel conditions, optimize the network hierarchy, and further reduce the calculation delay. At the same time, when calculating the stop probability, the differences in image blocks are taken into account, which reduces redundant calculations, improves the calculation efficiency in the communication process, and further reduces the calculation delay.

[0100] The image semantic communication device provided by the present invention is described below. The image semantic communication device described below and the image semantic communication method described above can be referenced to each other.

[0101] Reference Figure 6 , the image semantic communication device provided by the embodiment of the present invention includes: An image segmentation module 10 is used to segment the target image to be sent to obtain a plurality of image blocks; A feature extraction module 20, configured to extract features from each of the image blocks using an encoder to obtain semantic information of the target image; The semantic sending module 30 is used to input the semantic information into the first linear mapping layer for linear mapping processing, and then send it to the receiving end through the wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses the decoder to perform semantic recovery and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0102] In one embodiment, the feature extraction module 20 is further used to: Using each recursive transformer unit in the encoder to extract features of the image blocks layer by layer to obtain semantic information of the target image; wherein any of the recursive transformer units includes a visual transformer layer, and the visual transformer layer includes a first normalization layer, a multi-head self-attention layer, a second normalization layer and a multi-layer perceptron layer; Based on a preset depth adjustment strategy, when extracting features from the image block at any target layer, an image stop mark of the image block at the target layer is generated based on a preset stop function; the target layer is any one of the visual changer layers; Determining whether it is necessary to continue to perform feature extraction processing on the image blocks according to the image stop mark; If it is necessary to continue to perform feature extraction processing on the image block, the image block output by the target layer is input into the subsequent layer of the target layer, and feature extraction processing is continued on the image block until it is determined according to the image stop mark that feature extraction processing does not need to be continued on the image block, and the semantic information of the image block is obtained.

[0103] In one embodiment, the feature extraction module 20 is further used to: Based on a preset stopping function, calculating a target stopping score of the image block at the target layer; Obtaining a cumulative stop score of the image block; the cumulative stop score is obtained by accumulating the stop scores of the image block in each visual changer layer before the target layer; Adding the target stop score to the cumulative stop score to obtain a target cumulative stop score of the image block at the target layer; Determine the target element corresponding to the image block in the preset marking vector. If the target cumulative stop score is greater than or equal to the preset stop threshold, set the target element to the first eigenvalue. If the target cumulative stop score is less than the preset stop threshold, set the target element to the second eigenvalue to obtain the image stop mark of the image block in the target layer.

[0104] In one embodiment, the feature extraction module 20 is further used to: The image blocks output by the target layer are input into the subsequent layers of the target layer, and feature extraction processing is continued on the image blocks until it is determined according to the image stop mark that the image blocks do not need to be further subjected to feature extraction processing, and the stop scores and residual scores of the image blocks in each target visual changer layer are obtained; wherein the residual score of the Lth target visual changer layer is calculated based on the accumulated stop scores in the previous L-1 target visual changer layers, and the target visual changer layer is a visual changer layer that has subjected the image blocks to feature extraction processing; and L is a positive integer greater than 1; The weight value of the target visual changer layer is calculated according to the stop score and the remaining score of any of the target visual changer layers, and the semantic features of the image blocks output by each of the target visual changer layers are weighted and summed based on the weight value to obtain the semantic information of the image blocks.

[0105] In one embodiment, the image semantic communication apparatus further comprises a joint training module for: Acquire sample images, and divide each of the sample images into a plurality of sample image blocks; A sample data set is constructed based on the sample image blocks, and the encoder and the decoder are jointly trained using the sample data set to obtain an optimal combination of the number of recursive transformer units in the encoder and the decoder, as well as strategy parameters of a preset depth adjustment strategy.

[0106] In one embodiment, the joint training module is further used to: Initializing a first number of recursive transformer units in the encoder and a second number of recursive transformer units in a decoder of the receiving end; Based on a preset first number of iterations, the sample data set is input into an encoder for feature extraction to obtain sample semantic information of each of the sample images in the sample data set; The sample semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the sample semantic information into the second linear mapping layer for linear mapping processing, and then inputs it into the decoder for semantic restoration and image reconstruction to obtain a reconstructed sample image; Calculating the sample similarity between the reconstructed sample image and the sample image; Calculating a first computing delay of the transmitting end based on the computing capability of the transmitting end and the first number, and calculating a second computing delay of the receiving end based on the computing capability of the receiving end and the second number; If the sample similarity is greater than or equal to a preset similarity threshold, updating the first number and the second number according to the first calculation delay and the second calculation delay; determining an optimal combination of the number of recursive variator units in the encoder and the decoder according to the updated first number and second number; Based on the optimal combination, a preset depth adjustment strategy is added to the encoder and the decoder, and based on a preset second number of iterations, the encoder and the decoder are jointly trained using the sample data set to obtain strategy parameters of the depth adjustment strategy; the strategy parameters are used for the encoder and the decoder to calculate the stopping score of each visual transformer layer of each recursive transformer unit.

[0107] In one embodiment, the image semantic communication apparatus further includes a position embedding module, which is used to: Expanding each of the image blocks into an embedding vector; the vector dimension of the embedding vector is determined according to the size of the image block; The position information of the image blocks is embedded in the embedding vector; the position information is used to indicate the position of each of the image blocks in the target image.

[0108] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the steps of the image semantic communication method, for example, including: Segment the target image to be sent to obtain multiple image blocks; Using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; The semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0109] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0110] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can perform the steps of the image semantic communication method provided by the above methods, for example, including: Segment the target image to be sent to obtain multiple image blocks; Using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; The semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0111] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image semantic communication method provided by the above methods, for example, including: Segment the target image to be sent to obtain multiple image blocks; Using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; The semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image semantic communication method, applied to a sending end, characterized in that: The image semantic communication method comprises: Segment the target image to be sent to obtain multiple image blocks; Using an encoder to extract features from each of the image blocks to obtain semantic information of the target image; The semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic restoration and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

2. The image semantic communication method according to claim 1, characterized in that: The step of extracting features from each of the image blocks using an encoder to obtain semantic information of the target image includes: Using each recursive transformer unit in the encoder to extract features of the image blocks layer by layer to obtain semantic information of the target image; wherein any of the recursive transformer units includes a visual transformer layer, and the visual transformer layer includes a first normalization layer, a multi-head self-attention layer, a second normalization layer and a multi-layer perceptron layer; Based on a preset depth adjustment strategy, when extracting features from the image block at any target layer, an image stop mark of the image block at the target layer is generated based on a preset stop function; the target layer is any one of the visual changer layers; Determining whether it is necessary to continue to perform feature extraction processing on the image blocks according to the image stop mark; If it is necessary to continue to perform feature extraction processing on the image block, the image block output by the target layer is input into the subsequent layer of the target layer, and feature extraction processing is continued on the image block until it is determined according to the image stop mark that feature extraction processing does not need to be continued on the image block, and the semantic information of the image block is obtained.

3. The image semantic communication method according to claim 2, characterized in that: The step of generating an image stop mark of the image block at the target layer based on a preset stop function includes: Based on a preset stopping function, calculating a target stopping score of the image block at the target layer; Obtaining a cumulative stop score of the image block; the cumulative stop score is obtained by accumulating the stop scores of the image block in each visual changer layer before the target layer; Adding the target stop score to the cumulative stop score to obtain a target cumulative stop score of the image block at the target layer; Determine the target element corresponding to the image block in the preset marking vector. If the target cumulative stop score is greater than or equal to the preset stop threshold, set the target element to the first eigenvalue. If the target cumulative stop score is less than the preset stop threshold, set the target element to the second eigenvalue to obtain the image stop mark of the image block in the target layer.

4. The image semantic communication method according to claim 3, characterized in that: The step of inputting the image blocks output by the target layer into a subsequent layer of the target layer, and continuing to perform feature extraction processing on the image blocks until it is determined according to the image stop mark that it is not necessary to continue to perform feature extraction processing on the image blocks, and obtaining semantic information of the image blocks includes: The image blocks output by the target layer are input into the subsequent layers of the target layer, and feature extraction processing is continued on the image blocks until it is determined according to the image stop mark that the image blocks do not need to be further subjected to feature extraction processing, and the stop scores and residual scores of the image blocks in each target visual changer layer are obtained; wherein the residual score of the Lth target visual changer layer is calculated based on the accumulated stop scores in the previous L-1 target visual changer layers, and the target visual changer layer is a visual changer layer that has subjected the image blocks to feature extraction processing; and L is a positive integer greater than 1; The weight value of the target visual changer layer is calculated according to the stop score and the remaining score of any of the target visual changer layers, and the semantic features of the image blocks output by each of the target visual changer layers are weighted and summed based on the weight value to obtain the semantic information of the image blocks.

5. The image semantic communication method according to claim 1, characterized in that: Before the encoder is used to extract features from each of the image blocks to obtain semantic information of the target image, the method further includes: Acquire sample images, and divide each of the sample images into a plurality of sample image blocks; A sample data set is constructed based on the sample image blocks, and the encoder and the decoder are jointly trained using the sample data set to obtain an optimal combination of the number of recursive transformer units in the encoder and the decoder, as well as strategy parameters of a preset depth adjustment strategy.

6. The image semantic communication method according to claim 5, characterized in that: The method of jointly training the encoder and the decoder using the sample data set to obtain an optimal combination of the number of recursive transformer units in the encoder and the decoder, and strategy parameters of a preset depth adjustment strategy, includes: Initializing a first number of recursive transformer units in the encoder and a second number of recursive transformer units in a decoder of the receiving end; Based on a preset first number of iterations, the sample data set is input into an encoder for feature extraction to obtain sample semantic information of each of the sample images in the sample data set; The sample semantic information is input into the first linear mapping layer for linear mapping processing, and then sent to the receiving end through a wireless channel; the receiving end inputs the sample semantic information into the second linear mapping layer for linear mapping processing, and then inputs it into the decoder for semantic restoration and image reconstruction to obtain a reconstructed sample image; Calculating the sample similarity between the reconstructed sample image and the sample image; Calculating a first computing delay of the transmitting end based on the computing capability of the transmitting end and the first number, and calculating a second computing delay of the receiving end based on the computing capability of the receiving end and the second number; If the sample similarity is greater than or equal to a preset similarity threshold, updating the first number and the second number according to the first calculation delay and the second calculation delay; determining an optimal combination of the number of recursive variator units in the encoder and the decoder according to the updated first number and second number; Based on the optimal combination, a preset depth adjustment strategy is added to the encoder and the decoder, and based on a preset second number of iterations, the encoder and the decoder are jointly trained using the sample data set to obtain strategy parameters of the depth adjustment strategy; the strategy parameters are used for the encoder and the decoder to calculate the stopping score of each visual transformer layer of each recursive transformer unit.

7. The image semantic communication method according to claim 1, characterized in that: After the target image to be sent is segmented to obtain a plurality of image blocks, the method further includes: Expanding each of the image blocks into an embedding vector; the vector dimension of the embedding vector is determined according to the size of the image block; The position information of the image blocks is embedded in the embedding vector; the position information is used to indicate the position of each of the image blocks in the target image.

8. An image semantic communication device, characterized in that: include: An image segmentation module is used to segment the target image to be sent to obtain multiple image blocks; A feature extraction module, used to extract features from each of the image blocks using an encoder to obtain semantic information of the target image; A semantic sending module, used for inputting the semantic information into the first linear mapping layer for linear mapping processing, and then sending it to the receiving end through a wireless channel; the receiving end inputs the received semantic information into the second linear mapping layer for linear mapping processing, and then uses a decoder to perform semantic recovery and image reconstruction to obtain an output image; Among them, the encoder and the decoder have multiple recursive transformer units with the same structure, and the number of the recursive transformer units is determined by jointly training the encoder and the decoder with the computational delay as the optimization target and the similarity between the input initial sample image and the output sample image reconstructed by the receiving end as the constraint.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the image semantic communication method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image semantic communication method according to any one of claims 1 to 7 are implemented.

Images