Semantic communication image processing method and system, device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of communications, and provide a semantic communication image processing method and system, a device, and a storage medium. The method comprises: extracting a semantic feature vector of a source image; performing quantization and modulation to obtain a semantic feature signal; obtaining a transmission signal on the basis of transmission power, channel state information, and the semantic feature signal; and transmitting the transmission signal to a base station. Upon reception of a total received signal composed of a plurality of transmission signals, the base station performs channel equalization on the total received signal to obtain an estimated signal of each transmitting end, and inputs the estimated signal into a semantic reasoning model corresponding to the transmitting end for semantic reasoning to obtain target image processing data corresponding to the source image. A feature domain is divided into a plurality of semantic feature subspaces, such that semantic feature signals of a plurality of users are coded and transmitted in the semantic feature subspaces obtained by division, wherein the semantic feature signals are approximately orthogonal to each other, thereby reducing the processing difficulty of the total received signal by the base station and improving the processing efficiency of the base station, and thus improving the communication transmission efficiency.

Description

Semantic communication image processing method, system, device and storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a semantic communication image processing method, system, device and storage medium. BACKGROUND

[0002] Orthogonal Multiple Access (OMA) is a multi-user access method in a wireless communication system, which considers orthogonal allocation of time domain, frequency domain, spatial domain, power domain and other physical resources to realize multi-user differentiation, ensure signal orthogonality between different users, and effectively avoid multi-user interference. However, the user capacity of each orthogonal channel of the current orthogonal multiple access technology has approached the Shannon limit, and the number of user access is limited.

[0003] In the related art, a complex algorithm is set in the base station to distinguish the signals of different users. In this case, the complexity of the base station receiver is high, and the increase in the number of users will exponentially increase the signal interference between users, further increasing the processing difficulty of the receiver. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a semantic communication image processing method, system, device and storage medium, which can reduce the processing difficulty of the base station, improve the processing efficiency of the base station, and improve the communication transmission efficiency.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a semantic communication image processing method applied to a transmitting end, the method comprising:

[0006] extracting a semantic feature vector of a source image by using a feature extraction unit;

[0007] quantizing the semantic feature vector to obtain a discrete semantic feature vector, and modulating the discrete semantic feature vector by using a digital modulation unit to obtain a semantic feature signal;

[0008] obtaining a transmitting signal according to the transmitting power, channel state information corresponding to the transmitting end, and the semantic feature signal;

[0009] transmitting the transmitting signal to a base station, so that the base station receives a total receiving signal composed of the transmitting signals of a plurality of different transmitting ends, performs channel equalization on the total receiving signal to obtain an estimated signal corresponding to each transmitting end, and inputs the estimated signal into a semantic reasoning model corresponding to the transmitting end which is pre-trained to perform semantic reasoning, thereby obtaining target image processing data corresponding to the source image.

[0010] In some embodiments, the quantizing the semantic feature vector obtains a discrete semantic feature vector, including:

[0011] inputting the semantic feature vector into a linear layer to obtain a linear vector corresponding to a quantization bit number;

[0012] quantizing values greater than or equal to zero in the linear vector to one and otherwise to zero to obtain the discrete semantic feature vector.

[0013] In some embodiments, the obtaining a transmission signal according to the transmission power corresponding to each of the transmission ends, channel state information and the semantic feature signal includes:

[0014] obtaining a conjugate transpose matrix of the channel state information;

[0015] obtaining a transmission parameter corresponding to each of the transmission ends according to the transmission power and the conjugate transpose matrix;

[0016] obtaining the transmission signal according to a product of the transmission parameter and the semantic feature signal, and the total received signal is obtained by accumulating the transmission signal and a noise vector.

[0017] In some embodiments, the source image includes a to-be-classified image or a to-be-reconstructed image;

[0018] when the source image is the to-be-classified image, the semantic inference model is a neural network model, and the target image processing data is a classification result of the to-be-classified image;

[0019] when the source image is the to-be-reconstructed image, the feature extraction unit is an encoder unit, the semantic inference model is a decoder unit, and the target image processing data is a reconstruction result corresponding to the to-be-reconstructed image.

[0020] In some embodiments, when the feature extraction unit is an encoder unit, the extracting a semantic feature vector of a source image by using the feature extraction unit includes:

[0021] after first downsampling and self-attention processing of the source image by using a first encoding block, a first feature map is obtained, and a width and a height of the first feature map are each half of the source image;

[0022] after second downsampling and self-attention processing of the first feature map by using a second encoding block, a second feature map is obtained, and a width and a height of the second feature map are each half of the first feature map;

[0023] after third downsampling and self-attention processing of the second feature map by using a third encoding block, the semantic feature vector is obtained by encoding.

[0024] To achieve the above object, a second aspect of the embodiments of the present application provides a semantic communication image processing method applied to a base station, the method comprising:

[0025] obtaining a total received signal, the total received signal being obtained according to a transmission signal of at least one of the transmission ends, the transmission signal being obtained according to the semantic communication image processing method of any one of the first aspect;

[0026] performing channel equalization on the total received signal to obtain an estimated signal corresponding to each transmission end;

[0027] inputting the estimated signal into a pre-trained semantic inference model corresponding to the transmission end to perform semantic inference, to obtain target image processing data corresponding to the source image.

[0028] In some embodiments, the performing channel equalization on the total received signal to obtain an estimated signal corresponding to each transmission end comprises:

[0029] obtaining an estimated coefficient according to the channel state information and the conjugate transpose matrix;

[0030] obtaining the estimated signal according to a product of the estimated coefficient and the total received signal.

[0031] In some embodiments, when the semantic inference model is a neural network model, before the inputting the estimated signal into a pre-trained semantic inference model corresponding to the transmission end to perform semantic inference, the method further comprises:

[0032] obtaining a training data set, the training data set comprising a total sample signal and an image processing label, the total sample signal being obtained from a transmission sample signal generated according to a sample image by each of the transmission ends;

[0033] obtaining an estimated sample signal corresponding to each of the transmission ends according to the total sample signal;

[0034] performing joint training on the semantic inference model using the estimated sample signal to obtain an image prediction result;

[0035] calculating a first loss value according to the image processing label and the image prediction result, a second loss value according to the transmission sample signal and the total sample signal, and a third loss value according to the transmission sample signal and the sample image;

[0036] obtaining a total loss value according to the first loss value, the second loss value and the third loss value, adjusting the weight of the semantic inference model based on the total loss value to obtain a trained semantic inference model corresponding to each of the transmission ends.

[0037] In some embodiments, when the semantic inference model is a decoder unit, before the inputting the estimated signal into the pre-trained semantic inference model corresponding to the transmitting end for semantic inference, the method further comprises:

[0038] obtaining a training data set, the training data set comprising a total sample signal and an image processing label, the total sample signal being obtained from a transmitting sample signal generated according to a sample image by each of the transmitting ends;

[0039] obtaining an estimated sample signal corresponding to each of the transmitting ends according to the total sample signal;

[0040] jointly training the semantic inference model using the estimated sample signal to obtain an image prediction result;

[0041] calculating a total loss value according to the image processing label and the image prediction result;

[0042] adjusting the weight of the semantic inference model based on the total loss value to obtain a pre-trained semantic inference model corresponding to each of the transmitting ends.

[0043] To achieve the above object, a third aspect of embodiments of the present application proposes a semantic communication image processing system, comprising:

[0044] a plurality of transmitting ends, the transmitting ends being configured to obtain corresponding transmitting signals according to the semantic communication image processing method of any one of the first aspect;

[0045] a base station, the base station being configured to obtain a total received signal obtained according to the transmitting signals, and obtain target image processing data corresponding to each of the transmitting ends according to the semantic communication image processing method of any one of the second aspect.

[0046] To achieve the above object, a fourth aspect of embodiments of the present application proposes an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect or the second aspect when executing the computer program.

[0047] To achieve the above object, a fifth aspect of embodiments of the present application proposes a storage medium, the storage medium being a storage medium, the storage medium storing a computer program, and the computer program implementing the method of the first aspect or the second aspect when executed by a processor.

[0048] The semantic communication image processing method, system, device and storage medium provided in the embodiments of the present application utilize the feature extraction unit to extract the semantic feature vector of the source image, quantize the semantic feature vector to obtain a discrete semantic feature vector, and utilize the digital modulation unit to modulate the discrete semantic feature vector to obtain a semantic feature signal, obtain a transmission signal according to the transmission power, channel state information and semantic feature signal corresponding to the transmission end, transmit the transmission signal to the base station, so that the base station receives a total received signal formed by the transmission signals of multiple different transmission ends, performs channel equalization on the total received signal to obtain an estimated signal corresponding to each transmission end, and inputs the estimated signal into a semantic inference model corresponding to the transmission end and pre-trained to perform semantic inference, to obtain target image processing data corresponding to the source image. The embodiments of the present application separate the feature domain into multiple semantic feature subspaces by utilizing the feature extraction unit corresponding to different transmission ends to extract the semantic feature signal of the source image, realize encoding and transmission of the semantic feature signals of multiple users in the separated semantic feature subspaces, and the discrete semantic feature vectors of different users form the total received signal, but are approximately orthogonal to each other, thereby reducing the processing difficulty of the base station on the total received signal, improving the processing efficiency of the base station, and improving the communication transmission efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a schematic diagram of a semantic communication image processing system provided by an embodiment of the present application.

[0050] FIG. 2 is another schematic diagram of a semantic communication image processing system provided by an embodiment of the present application.

[0051] FIG. 3 is an optional flowchart of a semantic communication image processing method provided by an embodiment of the present application.

[0052] FIG. 4 is a flowchart of extracting a semantic feature vector of a source image by utilizing a feature extraction unit provided by an embodiment of the present application.

[0053] FIG. 5 is a structural schematic diagram of an encoder unit in the embodiments of the present application.

[0054] FIG. 6 is a flowchart of obtaining a transmission signal according to the transmission power, channel state information and semantic feature signal corresponding to the transmission end provided by an embodiment of the present application.

[0055] FIG. 7 is an optional flowchart of a semantic communication image processing method provided by an embodiment of the present application.

[0056] FIG. 8 is a schematic diagram of a training process of a semantic inference model provided by an embodiment of the present application.

[0057] FIG. 9 is a structural schematic diagram of a decoder unit provided by an embodiment of the present application.

[0058] FIG. 10 is another schematic diagram of the semantic communication image processing system according to an embodiment of the present application.

[0059] FIG. 11 is another schematic diagram of the training process of the semantic reasoning model according to an embodiment of the present application.

[0060] FIG. 12 is a schematic diagram of the classification performance of the image to be classified according to an embodiment of the present application.

[0061] FIG. 13 is another schematic diagram of the classification performance of the image to be classified according to an embodiment of the present application.

[0062] FIG. 14 is a schematic diagram of the reconstruction performance of the image to be reconstructed according to an embodiment of the present application.

[0063] FIG. 15 is another schematic diagram of the reconstruction performance of the image to be reconstructed according to an embodiment of the present application.

[0064] FIG. 16 is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0066] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the present application is only for the purpose of describing the embodiments of the present application and is not intended to limit the present application.

[0068] First, several terms involved in the present application are analyzed:

[0069] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0070] With the rapid development of large-scale wireless communication and the increasing demand for intelligent processing, various emerging intelligent services based on wireless communication technology have emerged, such as augmented reality, virtual reality, holographic communication, intelligent connected vehicles, etc., leading to a significant increase in the number of connected devices worldwide, which poses new challenges to communication technology. From 1G to 5G, mobile communication systems are guided by Shannon information theory, although the transmission rate has been greatly improved, but the communication system is facing the problems of gradually approaching the Shannon limit of system capacity, significant energy consumption of communication system and scarcity of high-quality spectrum resources.

[0071] In addition, Orthogonal Multiple Access (OMA) is a multi-user access method in a wireless communication system, which considers the orthogonal allocation of time domain, frequency domain, spatial domain, power domain and other physical resources to realize multi-user differentiation and ensure the orthogonality of signals between different users, effectively avoiding interference between multiple users. However, the user capacity of each orthogonal channel (time, frequency, subspace, sequence, etc.) of the current orthogonal multiple access technology has approached the Shannon limit, and the number of user access is limited, which makes it difficult to meet the challenges faced by communication technology. Therefore, 6G communication research needs to explore new theories and suitable multiple access technologies for information transmission to achieve efficient and flexible connection of massive users to the network and transmission of data on given wireless resources, thereby meeting the growing communication needs.

[0072] In the related art, complex algorithms are set in the base station to distinguish the signals of different users, in which case the base station receiver complexity is high, and the increase in the number of users will exponentially increase the signal interference between users, further increasing the processing difficulty of the receiver, resulting in the current orthogonal multiple access communication facing the problems of a large amount of data redundancy, high system complexity, limited capacity, high energy consumption and low information transmission efficiency.

[0073] Based on this, the embodiment of the application provides a semantic communication image processing method, system, device and storage medium, the semantic feature signals of the source image are extracted through the feature extraction units corresponding to different transmitting ends, the feature domain is separated into a plurality of semantic feature subspaces, the semantic feature signals of a plurality of users are encoded and transmitted in the separated semantic feature subspaces, the discrete semantic feature vectors of different users constitute a total received signal, but are approximately orthogonal to each other, thereby reducing the processing difficulty of the base station for the total received signal, improving the processing efficiency of the base station, and improving the communication transmission efficiency.

[0074] The embodiment of the application provides a semantic communication image processing method, system, device and storage medium, and the method is specifically described as follows.

[0075] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0076] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0077] The semantic communication image processing method provided by the embodiments of the present application relates to the technical field of communication. The semantic communication image processing method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be a computer program running in the terminal or the server. For example, the computer program can be a native program or a software module in an operating system; can be a native application program (APP), that is, a program that needs to be installed in an operating system to run, such as a client supporting semantic communication image processing, and can also be a small program, that is, a program that only needs to be downloaded into a browser environment to run; and can also be a small program that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in. The terminal communicates with the server through a network. The semantic communication image processing method can be executed by the terminal or the server, or cooperatively executed by the terminal and the server.

[0078] In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch or the like. The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms; or a service node in a blockchain system, the service nodes in the blockchain system form a peer-to-peer (P2P) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP) protocol. The terminal and the server can be connected through communication connection modes such as Bluetooth, universal serial bus (USB) or network, and the embodiments are not limited here.

[0079] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0080] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards of the country or region. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0081] First, the semantic communication image processing system provided by the embodiments of the present application is described.

[0082] Referring to FIG. 1, FIG. 1 is a schematic diagram of a semantic communication image processing system provided by an embodiment of the present application. The semantic communication image processing system in FIG. 1 is a multi-user semantic multiple access network based on single-carrier frequency-division multiple access (SFDMA). It includes multiple transmitting ends and a base station, and each transmitting end can be regarded as a user, for example, N users are taken as an example for illustration in FIG. 1.

[0083] Each user corresponds to a semantic encoder, and the parameters of the semantic encoder are denoted as φ i for the i th transmitting end. i The semantic feature signal x i can be obtained according to the user's source image u i, and is denoted as: i i The channel fading coefficient g i between the i th transmitting end and the base station is denoted as:​​i As channel state information, semantic feature signal x is obtained based on the channel state information. i The corresponding transmitted signal. Since each user's semantic encoder is different, each semantic feature signal corresponds to a semantic feature subspace.

[0084] Then, the base station's receiver simultaneously receives transmitted signals from multiple users. At this point, the transmitted signals are superimposed, forming the total received signal y. The base station includes semantic decoders corresponding one-to-one with the transmitters. These semantic decoders correspond to semantic encoders, and the base station uses the semantic decoders to obtain the source image u from the total received signal y. i Corresponding target image processing data

[0085] In one embodiment, referring to FIG2, FIG2 is another schematic diagram of the semantic communication image processing system provided in the embodiments of this application. Referring to FIG1, the semantic encoder at the transmitting end in FIG2 includes a feature extraction unit, a quantizer, and a digital modulation unit, and the semantic decoder at the base station includes an equalizer and a semantic inference model.

[0086] Taking the i-th transmitter as an example, the transmitter uses a feature extraction unit to extract information from the user's source image u. i The semantic feature vector z is obtained from i Then the quantizer Q processes the semantic feature vector z i Quantization is performed to obtain discrete semantic feature vector b. i Then, the discrete semantic feature vector b is processed by the digital modulation unit. i Modulation is performed to obtain semantic feature signal x i .

[0087] In addition, correspondingly, after the base station obtains the total received signal y, it uses an equalizer (EQ) to perform channel equalization on the total received signal y to obtain the estimated signal for each transmitter. For example, the estimated signal for the i-th transmitter is... Then, the semantic reasoning model corresponding to the i-th transmitter is used. Semantic reasoning is performed on the estimated signal to obtain target image processing data. Where, θ i Representation of semantic reasoning model Network parameters.

[0088] The semantic communication image processing method applied to the transmitting end in the embodiments of this application is described below with reference to Figures 1 and 2.

[0089] Figure 3 is an optional flowchart of the semantic communication image processing method provided in an embodiment of this application. The method in Figure 3 may include, but is not limited to, steps 110 to 140. It is also understood that this embodiment does not specifically limit the order of steps 110 to 140 in Figure 3, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0090] Step 110: Extract the semantic feature vector of the source image using the feature extraction unit.

[0091] In one embodiment, due to different image processing tasks, the source image includes an image to be classified or an image to be reconstructed. The image to be classified is used for image classification, and the image to be reconstructed is used for image reconstruction. When the source image is an image to be classified, the semantic reasoning model is a neural network model, and the target image processing data is the classification result of the image to be classified. When the source image is an image to be reconstructed, the feature extraction unit is an encoder unit, the semantic reasoning model is a decoder unit, and the target image processing data is the reconstruction result corresponding to the image to be reconstructed.

[0092] In one embodiment, when the source image is an image to be classified, for the i-th transmitter, the feature extraction unit It can be a neural network model, capable of analyzing source images u i Features are extracted and encoded to obtain the semantic feature vector z. i This feature extraction process can be represented as:

[0093] In one embodiment, when the source image is an image to be reconstructed, the feature extraction unit is an encoder unit. Referring to Figure 4, Figure 4 is a flowchart of extracting semantic feature vectors of a source image using a feature extraction unit according to an embodiment of this application, specifically including:

[0094] Step 410: After performing the first downsampling and self-attention processing on the source image using the first coding block, the first feature map is obtained.

[0095] In one embodiment, when the source image is the image to be reconstructed, the source image is represented as: Where H and W represent the height and width of the source image, and 3 represents the number of channels of the source image.

[0096] At this time, refer to FIG. 5, which is a structural schematic diagram of an encoder unit in an embodiment of the present application. The encoder unit in FIG. 5 includes three encoding blocks, namely a first encoding block, a second encoding block and a third encoding block. The first encoding block, the second encoding block and the third encoding block are all Swin Transformer structures, each including an encoding patch block and a Swin Transformer encoding block. The encoding patch block is used to divide the input image in size to obtain a plurality of non-overlapping patches, which are then sent to the Swin Transformer encoding block. The Swin Transformer encoding block is used to perform self-attention calculation within each patch, and then integrates information through a shift operation, thereby realizing the encoding process of the input image. The Swin Transformer structure uses the encoding patch block and the Swin Transformer encoding block to jointly extract and convert semantic information in the image.

[0097] In an embodiment, the first encoding block first performs first downsampling on the source image u i to divide the source image into non-overlapping patches (Patch). These patches are labeled in the order from top left to bottom right to obtain a serial sequence, which is embedded into Embedding, and then subjected to self-attention processing by the Swin Transformer block to obtain a first feature map. The size of the first feature map is C is equal to 3, which is the number of channels. The width and height of the first feature map are each half of the source image, and the number of channels is consistent with the source image.

[0098] Step 420: After the second encoding block is used to perform second downsampling and self-attention processing on the first feature map, a second feature map is obtained.

[0099] In an embodiment, in the second encoding block, the first feature map is first subjected to second downsampling by the encoding patch block to reduce the size of the first feature map and reduce the number of patches, thereby reducing the image resolution and adjusting the number of channels, which helps to reduce the subsequent calculation amount. After the patch processing, the height and width of the feature map are each reduced to half of the original, while the number of channels is expanded to twice the original. Subsequently, the adjusted patch is again processed by the Swin Transformer encoding block to output a second feature map. The size of the second feature map is The width and height of the second feature map are each half of the first feature map, and the number of channels is twice that of the first feature map.

[0100] Step 430: After the third encoding block is used to perform third downsampling and self-attention processing on the second feature map, a semantic feature vector is obtained.

[0101] In an embodiment, the second feature map continues to be down-sampled by a third encoding patch block, and then is processed by a Swin Transformer encoding block to obtain a final feature map, which is denoted as with a width and a height each being half of the second feature map and a channel number being twice of the second feature map. Finally, the feature map is encoded to output a semantic feature vector The semantic feature vector contains key semantic information in the source image and has a significantly reduced size compared with the source image.

[0102] Overall, the feature extraction unit i.e., the encoder unit, extracts the semantic feature vector of the source image, and the process is denoted as

[0103] Step 120: quantize the semantic feature vector to obtain a discrete semantic feature vector, and modulate the discrete semantic feature vector by using a digital modulation unit to obtain a semantic feature signal.

[0104] In an embodiment, in order to improve the transmission efficiency of the communication system and simplify the processing process of the base station, the embodiment of the present application performs binary quantization on the semantic feature vector z i , and uses a quantizer to quantize the semantic feature vector to obtain a discrete semantic feature vector. The specific process includes: inputting the semantic feature vector into a linear layer to obtain a linear vector corresponding to the quantization bit number, quantizing the values greater than or equal to zero in the linear vector to one, and otherwise quantizing to zero to obtain a discrete semantic feature vector.

[0105] In an embodiment, the quantization bit number d has a certain relationship with the image processing result, for example, d is 16 or 64. Taking d as 64 as an example, assuming that the semantic feature vector z i has a dimension of 1024, inputting the semantic feature vector into a linear layer to obtain a linear vector corresponding to the quantization bit number means that the 1024-dimensional semantic feature vector z i is converted into a 64-dimensional linear vector by using the linear layer.

[0106] Next, the linear vector is binary-discretized, i.e., the values greater than or equal to zero in the linear vector are quantized to one, and otherwise quantized to zero to obtain a discrete semantic feature vector. Here, the discrete semantic feature vector is a discrete vector composed of 0 or 1, and is denoted as

[0107] where b i represents the discrete semantic feature vector corresponding to the i-th transmitting end, T represents transposition, Q i () represents the quantization process by using 0 or 1, z i,j represents the numerical value in the linear vector.

[0108] In one embodiment, a discrete semantic feature vector b is obtained. i Then, a digital modulator is further utilized. For discrete semantic feature vector b i Modulation is performed to convert the bitstream into a signal form suitable for transmission over the channel, such as BPSK modulation. The modulated signal is then normalized to obtain the normalized semantic feature signal, represented as:

[0109] x i =[x i,1 ,x i,2 ,…,x i,d ] H =Norm(Θ m (b i ),i∈{1,..,N}

[0110] Where H represents the conjugate transpose, x i Let represent the semantic feature signal of the i-th transmitter. Norm(·) represents the normalization operation. The purpose of normalization is to ensure that the semantic feature signal can adapt to the characteristics of the channel during transmission and to ensure that the signal has a uniform power level, which helps the base station to accurately recover the transmitted semantic feature signal.

[0111] Step 130: Obtain the transmitted signal based on the transmit power, channel state information, and semantic feature signals corresponding to the transmitter.

[0112] In one embodiment, N transmitters simultaneously transmit corresponding semantic feature signals to the base station in the time-frequency domain, and transmit them via a multiple access communication network. For the i-th transmitter, g... i p represents the channel fading coefficient from the base station to the channel state information. i Given its transmission power, the process of obtaining its corresponding transmission signal is shown in Figure 6. Figure 6 is a flowchart of obtaining the transmission signal based on the transmission power, channel state information, and semantic feature signal of the transmitting end, provided by an embodiment of this application. Specifically, it includes:

[0113] Step 610: Obtain the conjugate transpose matrix of the channel state information.

[0114] Step 620: Obtain the transmission parameters corresponding to each transmitter based on the transmission power and the conjugate transpose matrix.

[0115] Step 630: Obtain the transmitted signal by multiplying the transmission parameters and the semantic feature signal.

[0116] The total received signal is obtained by summing the transmitted signal and the noise vector.

[0117] In an embodiment, the conjugate transpose matrix of the channel state information is The transmission parameter is represented as:

[0118] The i-th transmission signal is represented as:

[0119] Step 140: transmitting the transmission signal to the base station.

[0120] In an embodiment, at this time, the N transmission ends transmit the transmission signals in the same time-frequency domain, and therefore the total received signal y received by the base station is obtained by accumulating the transmission signals and the noise vector, and the total received signal y is represented as:

[0121] wherein n is a complex noise vector with a mean of 0 and a variance of δ 2

[0122] When the base station receives the total received signal composed of the transmission signals of multiple different transmission ends, the channel equalization is performed on the total received signal to obtain an estimated signal corresponding to each transmission end, and the estimated signal is input to the semantic inference model corresponding to the transmission end which is pre-trained to perform semantic inference, to obtain target image processing data corresponding to the source image.

[0123] Embodiments of the present application extract semantic feature signals of a source image through feature extraction units corresponding to different transmission ends, separate the feature domain into multiple semantic feature subspaces, realize encoding and transmission of semantic feature signals of multiple users in the separated semantic feature subspaces, and form a total received signal with discrete semantic feature vectors of different users, but the discrete semantic feature vectors of different users are approximately orthogonal to each other, thereby reducing the processing difficulty of the base station on the total received signal, improving the processing efficiency of the base station, and improving the communication transmission efficiency.

[0124] The semantic communication image processing method applied to the base station in the embodiments of the present application will be described below with reference to FIG. 1 and FIG. 2.

[0125] FIG. 7 is an optional flowchart of the semantic communication image processing method provided by the embodiments of the present application. The method in FIG. 7 can include, but is not limited to, steps 710 to 740. It can be understood that the order of steps 710 to 740 in FIG. 7 is not specifically limited in the embodiments, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0126] Step 710: obtaining a total received signal.

[0127] In an embodiment, the total received signal is obtained according to the transmission signal of at least one transmission end, and the transmission signal is obtained according to any one of the semantic communication image processing methods described in the above embodiments.​

[0128] Step 720: performing channel equalization on the total received signal to obtain an estimated signal corresponding to each transmitting end.

[0129] In an embodiment, the channel equalization is performed in an ideal state, that is, assuming that the channel state information between each transmitting end and the base station is known information. It can be understood that if the channel state information is unknown, channel estimation is needed at this time, and common channel estimation methods such as LS channel estimation, MMSE, LMMSE, etc. can be used, and the present embodiment does not limit this.

[0130] The process of performing channel equalization on the total received signal to obtain an estimated signal corresponding to each transmitting end specifically includes: obtaining an estimated coefficient according to the channel state information and the conjugate transpose matrix, and obtaining an estimated signal according to the product of the estimated coefficient and the total received signal.

[0131] The i-th estimated coefficient is represented as:

[0132] The i-th estimated signal is represented as:

[0133] The i-th estimated signal corresponding to the i-th transmitting signal can be obtained by performing transformation calculation on the estimated signal:

[0134] In the above formula, represents the estimated signal corresponding to the i-th transmitting end, represents noise in the Rayleigh fading channel, represents the interference signal of the transmitting signal of the i-th transmitting end due to the transmitting signals of other transmitting ends. Since the channel state information g i is known, in the channel equalization process, the total received signal can be divided by g i , so that the channel effect can be converted from multiplication operation to additive operation, thereby reducing the learning burden of the subsequent semantic inference model.

[0135] Step 730: inputting the estimated signal into a pre-trained semantic inference model corresponding to the transmitting end to perform semantic inference, to obtain target image processing data corresponding to the source image.

[0136] In an embodiment, when the source image is a to-be-classified image, the semantic inference model is a neural network model, and the target image processing data is a classification result of the to-be-classified image. When the source image is a to-be-reconstructed image, the feature extraction unit is an encoder unit, the semantic inference model is a decoder unit, and the target image processing data is a reconstruction result corresponding to the to-be-reconstructed image.

[0137] In an embodiment, after signal equalization processing, the i-th estimated signal is input into the semantic inference network The semantic inference network obtains semantic features therefrom, performs classification processing or image reconstruction processing thereon, and outputs a final result to obtain target image processing data corresponding to the i-th transmitting end The inference process is represented as:

[0138] wherein θ i represents parameters of the i-th semantic inference network.

[0139] The semantic inference models of different types of source images and their corresponding training processes are described below.

[0140] In an embodiment, when the source image is a to-be-classified image, the semantic inference model is a neural network model. Referring to FIG. 8, FIG. 8 is a schematic diagram of a training process of a semantic inference model according to an embodiment of the present application, which specifically includes the following steps:

[0141] Step 810: Obtain a training data set.

[0142] First, the corresponding transmitting sample signals are generated according to the sample images of each transmitting end, then the transmitting sample signals are accumulated, noise is superimposed to obtain a total sample signal, and the image processing label corresponding to each sample image is obtained, and the total sample signal and the image processing label are associated. In this way, multiple groups of total sample signals are generated.

[0143] In an embodiment, there are L samples in the training data set, corresponding to L total sample signals, and the sampling number of the semantic encoder of each transmitting end for the semantic feature signal is M, that is, for the semantic feature signal of the transmitting sample corresponding to each total sample signal, the semantic encoder divides the semantic feature signal into M discrete data points, and correspondingly, the transmitting sample signal also corresponds to M discrete transmitting data, and the image processing label also corresponds to M discrete label data. The sampling number herein will affect the complexity and accuracy of the base station data processing, and the sampling number is set according to actual needs in the present embodiment.

[0144] Step 820: Obtain an estimated sample signal corresponding to each transmitting end according to the total sample signal.

[0145] In an embodiment, the estimated sample signal corresponding to each transmitting end is obtained according to the total sample signal in the above-described manner of calculating the estimated signal.

[0146] Step 830: Jointly train the semantic inference model using the estimated sample signal to obtain an image prediction result.

[0147] In one embodiment, data from multiple transmitters are jointly trained to perform multi-user joint source-channel-destination coding, achieving the goal of encoding and transmitting semantically relevant information from multiple users in separate semantic feature domains. In other words, the estimated sample signals are input into the semantic inference model corresponding to each transmitter to obtain the corresponding image prediction results.

[0148] Step 840: Calculate the first loss value based on the image processing label and image prediction result, calculate the second loss value based on the transmitted sample signal and the total sample signal, and calculate the third loss value based on the transmitted sample signal and the sample image.

[0149] In one embodiment, the first loss value is expressed as:

[0150] Among them, y (l,m) This represents the discrete label data of the m-th image processing label corresponding to the transmitted sample of the i-th transmitter in the l-th sample, and represents the image prediction result of the i-th transmitter in the l-th sample. This indicates that the semantic reasoning model has parameters θ. i Under the premise that y (l,m) get The probability distribution is shown. Understandably, the first loss value here is used to measure the difference between the image processing label and the image prediction result, so the smaller the difference, the better.

[0151] The second loss value is expressed as:

[0152] in, Y represents the j-th dimension of the m-th discrete transmitted data corresponding to the transmitted sample signal of the i-th transmitter in the l-th sample. j Let the j-th dimension of the total sample signal be denoted as . express and Y j The information entropy between them. It can be understood that the optimization objective of the second loss value here is to maximize the semantic transmission rate, that is, to maximize the corresponding information entropy.

[0153] The third loss value is expressed as:

[0154] in, X represents the sample image of the i-th transmitter in the l-th sample. i,j This represents the j-th dimension of the transmitted sample signal from the i-th transmitter after quantization and modulation. X represents i,j and the information entropy between the first loss value and the second loss value. It can be understood that the optimization target of the third loss value here is to measure the semantic compression quality, and the higher the compression degree is, the smaller the corresponding information entropy is.

[0155] Step 850: obtaining a total loss value according to the first loss value, the second loss value and the third loss value, and performing weight adjustment on the semantic inference model based on the total loss value to obtain a trained semantic inference model corresponding to each transmitting end.

[0156] In an embodiment, the total loss value is expressed as:

[0157] wherein, represents the total loss value corresponding to the image to be classified, β i ≥ 0 represents a weighting parameter between the inference performance and the model robustness of the i-th transmitting end, which can be set according to actual requirements.

[0158] The design of the total loss value in the embodiments of the present application achieves a trade-off between the amount of transmitted semantic information, inter-user interference and inference accuracy, so that the weight adjustment is performed on the semantic inference model according to the total loss value until the preset training termination condition is reached, that is, the trained semantic inference model corresponding to each transmitting end is obtained. It can be understood that the weights of different semantic inference models are different and can adapt to different channel states of the transmitting end.

[0159] In an embodiment, when the source image is an image to be reconstructed, the semantic inference model is a decoder unit. Referring to FIG. 9, FIG. 9 is a structural schematic diagram of a decoder unit provided by an embodiment of the present application. In combination with FIG. 5, it can be known from FIG. 9 that the decoder unit has a symmetrical structure with the encoder unit, including three decoding blocks, which are a first decoding block, a second decoding block and a third decoding block. The first decoding block, the second decoding block and the third decoding block are all Swin Transformer structures, including a Swin Transformer decoding block and a decoding patch block. The Swin Transformer decoding block corresponds to the Swin Transformer encoding block, is used for decoding and reconstructing a signal, and the decoding patch block is used for up-sampling the input image. Therefore, in the embodiments of the present application, the estimated signal Firstly, the first decoding block is entered, and after reconstruction and up-sampling, the estimated signal becomes Then, the second decoding block is entered, and after reconstruction and up-sampling, the estimated signal becomes Finally, the third decoding block is entered, and after reconstruction and up-sampling, the estimated signal becomes HxWx3, at this time, the target image processing data is the reconstruction result corresponding to the image to be reconstructed The whole process is represented as:

[0160] wherein, represents the semantic reasoning network.

[0161] In an embodiment, corresponding to Fig. 2, when the source image is the image to be reconstructed, referring to Fig. 10, Fig. 10 is another schematic diagram of the semantic communication image processing system provided by the embodiment of the present application. The semantic encoder of the transmitting end includes a feature extraction unit, a quantizer and a digital modulation unit, the semantic decoder of the base station includes an equalizer and a semantic reasoning model, and the feature extraction unit is the encoder unit and the semantic reasoning model is the decoder unit.

[0162] The training process of the decoder unit is described below. In an embodiment, referring to Fig. 11, Fig. 11 is another schematic diagram of the training process of the semantic reasoning model provided by the embodiment of the present application, which specifically includes:

[0163] Step 1110: Obtain a training data set.

[0164] The training data set is similar to the generation method of the training data set in step 810. The training data set includes a total sample signal and an image processing label, and the total sample signal is obtained from the transmission sample signal of each transmitting end, and the transmission sample signal is generated according to a sample image.

[0165] Step 1120: Obtain an estimated sample signal corresponding to each transmitting end according to the total sample signal.

[0166] Step 1130: Jointly train the semantic reasoning model using the estimated sample signal to obtain an image prediction result.

[0167] Step 1140: Calculate a total loss value according to the image processing label and the image prediction result.

[0168] The total loss value is represented as:

[0169] wherein, represents the total loss value corresponding to the image to be reconstructed, E represents expectation, represents the joint probability distribution of the image processing label and the image prediction result, u i represents the image processing label, represents the image prediction result.

[0170] Step 1150: Adjust the weight of the semantic reasoning model based on the total loss value to obtain a trained semantic reasoning model corresponding to each transmitting end.

[0171] In an embodiment, the semantic inference model is weight-adjusted according to the total loss value until a preset training termination condition is reached, and thus a trained semantic inference model corresponding to each transmitting end is obtained. The training termination condition here can be that the trained semantic inference model satisfies an evaluation index MS-SSIM. MS-SSIM is a multi-scale structural similarity, which is an index for measuring the quality of image reconstruction, and the similarity between images is evaluated by considering the structural similarity at different scales. The calculation process of MS-SSIM includes dividing the image processing label and the image prediction result into sub-images of different scales, calculating the SSIM index for each scale of sub-image, the SSIM index being a structural similarity measure, and then performing a weighted average on these indexes to obtain the final MS-SSIM value. The value of MS-SSIM ranges from 0 to 1, and the closer the value is to 1, the higher the similarity between the reconstructed image and the original image, and the better the image quality. It can be understood that different semantic inference models have different weights and can adapt to different channel states of different transmitting ends.

[0172] In an embodiment, referring to FIG. 12, FIG. 12 is a schematic diagram of classification performance of an image to be classified according to an embodiment of the present application. In FIG. 12, image classification is performed for two users U1 and U2, where the quantization bit number is d = 16 and the quantization bit number d = 64, and the training signal-to-noise ratio is 5 dB. In the figure, the vertical axis represents the classification accuracy (percentage), and the horizontal axis represents the signal-to-noise ratio (dB). Upper bound represents the performance upper bound, DeepJSCC represents a deep joint source channel coding scheme in the related art, and SFDMA represents the semantic communication image processing method provided in the embodiment of the present application. As can be seen from FIG. 12, the classification accuracy of the SFDMA scheme provided in the embodiment of the present application is always better than that of the Deep JSCC scheme in the related art, and tends to approach the performance upper bound Upper bound corresponding to different users, proving the effectiveness of the semantic communication image processing method in the embodiment of the present application in image classification.

[0173] In an embodiment, referring to FIG. 13, FIG. 13 is another schematic diagram of classification performance of an image to be classified according to an embodiment of the present application. In the figure, it is shown that in the case of quantization bit number d = 128 and training signal-to-noise ratio 5 dB, handwritten digits of different users need to be recognized. According to the recognition result, the semantic feature signals of users U1, U2 and U3 are separated in the semantic feature subspace, and the classification accuracy of each user is above 94%.

[0174] In an embodiment, referring to FIG. 14, FIG. 14 is a schematic diagram of reconstruction performance of an image to be reconstructed according to an embodiment of the present application. FIG. 14 is a visualization of the semantic feature subspace. As can be seen from FIG. 14, the images of two users U1 and U2 can be separated in the semantic feature subspace, and the reconstruction process will not be disturbed by other users.

[0175] In an embodiment, referring to FIG. 15, FIG. 15 is another schematic diagram of reconstruction performance of a to-be-reconstructed image provided in the embodiment of the present application. The reconstruction performance comparison is shown in FIG. 15 when the peak signal-to-noise ratio (PSNR) and the multi-scale structural similarity (MS-SSIM) are respectively used as evaluation indexes. As can be seen from FIG. 15, whether the peak signal-to-noise ratio (PSNR) or the multi-scale structural similarity (MS-SSIM), the reconstruction performance of the SFDMA scheme provided in the embodiment of the present application is always superior to that of the DeepJSCC scheme in the related art, and tends to the performance upper bound, proving the effectiveness of the semantic communication image processing method in the embodiment of the present application in image reconstruction.

[0176] The embodiment of the present application extracts semantic features of a source image, and performs multi-user joint source-channel-destination coding in combination with quantization and coding, to realize coding and transmission of semantic feature signals of multiple users in separated semantic feature subspaces, effectively solve the interference between users, and improve the number of user access. Meanwhile, the semantic feature signals of different users are approximately orthogonal, and the semantic information of each user can be coded and transmitted in the separated feature subspaces, so that the semantic reasoning model is more robust, and the quality of image classification or image reconstruction is ensured.

[0177] The technical scheme provided in the embodiment of the present application extracts semantic feature vectors of a source image by using a feature extraction unit, quantizes the semantic feature vectors to obtain discrete semantic feature vectors, and modulates the discrete semantic feature vectors by using a digital modulation unit to obtain semantic feature signals. The transmission signal is obtained according to the corresponding transmission power, channel state information and semantic feature signals of the transmitting end, the total received signal formed by the transmission signals of multiple different transmitting ends is transmitted to the base station, the channel equalization is performed on the total received signal, the estimated signal corresponding to each transmitting end is obtained, and the estimated signal is input into the semantic reasoning model corresponding to the transmitting end and pre-trained to perform semantic reasoning, to obtain the target image processing data corresponding to the source image. In the embodiment of the present application, the semantic feature signals of the source image are extracted by using the feature extraction unit corresponding to different transmitting ends, the feature domain is separated into multiple semantic feature subspaces, the coding and transmission of the semantic feature signals of multiple users are realized in the separated semantic feature subspaces, the discrete semantic feature vectors of different users form the total received signal, but are approximately orthogonal to each other, thereby reducing the processing difficulty of the base station on the total received signal, improving the processing efficiency of the base station, and improving the communication transmission efficiency.

[0178] The embodiment of the present application also provides an electronic device, comprising:

[0179] at least one memory;

[0180] at least one processor;

[0181] at least one program;

[0182] The program is stored in the memory, and the processor executes the at least one program to implement the semantic communication image processing method described above. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, and the like.

[0183] Referring to FIG. 16, FIG. 16 illustrates a hardware structure of an electronic device according to another embodiment, which includes:

[0184] The processor 1601 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and the like, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0185] The memory 1602 can be implemented in a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and the like. The memory 1602 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1602 and are called and executed by the processor 1601 to implement the semantic communication image processing method of the embodiments of the present application.

[0186] The input / output interface 1603 is configured to implement information input and output.

[0187] The communication interface 1604 is configured to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, and the like).

[0188] The bus 1605 is configured to transmit information between various components (for example, the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604) of the device.

[0189] The processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604 are connected to each other through the bus 1605 to realize the communication connection between them inside the device.

[0190] The embodiment of the present application also provides a storage medium, the storage medium is a storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to realize the semantic communication image processing method.

[0191] The memory is a non-transitory storage medium and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0192] The semantic communication image processing method, system, device and storage medium provided by the embodiment of the present application can extract the semantic feature vector of the source image by using the feature extraction unit, quantize the semantic feature vector to obtain a discrete semantic feature vector, modulate the discrete semantic feature vector by using a digital modulation unit to obtain a semantic feature signal, obtain a transmission signal according to the transmission power, channel state information and semantic feature signal corresponding to the transmitting end, transmit the transmission signal to the base station, so that the base station receives a total received signal formed by the transmission signals of multiple different transmitting ends, performs channel equalization on the total received signal to obtain an estimated signal corresponding to each transmitting end, and inputs the estimated signal into a pre-trained semantic inference model corresponding to the transmitting end to perform semantic inference, thereby obtaining target image processing data corresponding to the source image. The embodiment of the present application extracts the semantic feature signal of the source image by using the feature extraction unit corresponding to different transmitting ends, separates the feature domain into multiple semantic feature subspaces, realizes the encoding transmission of the semantic feature signals of multiple users in the separated semantic feature subspaces, and the discrete semantic feature vectors of different users form the total received signal, but are approximately orthogonal to each other, thereby reducing the processing difficulty of the base station for the total received signal, improving the processing efficiency of the base station, and improving the communication transmission efficiency.

[0193] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0194] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps or different steps.

[0195] The apparatus embodiments described above are merely exemplary, and units described as separate components may or may not be physically separate, i.e., may be located in one place, or may be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0196] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0197] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0198] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0199] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0200] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0201] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0202] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various program storage media.

[0203] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A semantic communication image processing method, characterized in that, Applied to the transmitting end, the method includes: The semantic feature vector of the source image is extracted using a feature extraction unit; The semantic feature vector is quantized to obtain a discrete semantic feature vector, and the discrete semantic feature vector is modulated using a digital modulation unit to obtain a semantic feature signal; The transmitted signal is obtained based on the transmit power, channel state information, and semantic feature signal corresponding to the transmitter. The transmitted signal is transmitted to the base station so that the base station receives a total received signal composed of the transmitted signals from multiple different transmitters. Channel equalization is performed on the total received signal to obtain an estimated signal corresponding to each transmitter. The estimated signal is then input into a pre-trained semantic reasoning model corresponding to the transmitter for semantic reasoning to obtain the target image processing data corresponding to the source image.

2. The semantic communication image processing method according to claim 1, characterized in that, The process of quantizing the semantic feature vector to obtain a discrete semantic feature vector includes: The semantic feature vector is input into the linear layer to obtain the linear vector corresponding to the number of quantization bits; Values ​​greater than or equal to zero in the linear vector are quantized to one, and otherwise quantized to zero, to obtain the discrete semantic feature vector.

3. The semantic communication image processing method according to claim 1, characterized in that, The step of obtaining the transmitted signal based on the transmit power, channel state information, and semantic feature signal corresponding to the transmitting end includes: Obtain the conjugate transpose matrix of the channel state information; The transmission parameters corresponding to each transmitter are obtained based on the transmission power and the conjugate transpose matrix. The transmitted signal is obtained by multiplying the transmitted parameters and the semantic feature signal, and the total received signal is obtained by accumulating the transmitted signal and the noise vector.

4. The semantic communication image processing method according to claim 1, characterized in that, The source image includes an image to be classified or an image to be reconstructed. When the source image is an image to be classified, the semantic reasoning model is a neural network model, and the target image processing data is the classification result of the image to be classified. When the source image is an image to be reconstructed, the feature extraction unit is an encoder unit, the semantic reasoning model is a decoder unit, and the target image processing data is the reconstruction result corresponding to the image to be reconstructed.

5. The semantic communication image processing method according to claim 4, characterized in that, When the feature extraction unit is an encoder unit, the step of extracting the semantic feature vector of the source image using the feature extraction unit includes: After performing a first downsampling and self-attention processing on the source image using the first coding block, a first feature map is obtained, wherein the width and height of the first feature map are both half of the source image. After performing a second downsampling and self-attention processing on the first feature map using the second coding block, a second feature map is obtained, the width and height of which are both half of the first feature map; The semantic feature vector is obtained by performing a third downsampling and self-attention processing on the second feature map using a third coding block.

6. A semantic communication image processing method, characterized in that, Applied to a base station, the method includes: A total received signal is obtained based on a transmission signal from at least one of the transmitting ends, wherein the transmission signal is obtained by the semantic communication image processing method according to any one of claims 1 to 5; Channel equalization is performed on the total received signal to obtain the estimated signal corresponding to each transmitter; The estimated signal is input into a pre-trained semantic reasoning model corresponding to the transmitter to perform semantic reasoning, thereby obtaining the target image processing data corresponding to the source image.

7. The semantic communication image processing method according to claim 6, characterized in that, The step of performing channel equalization on the total received signal to obtain the estimated signal corresponding to each transmitter includes: The estimated coefficients are obtained based on the channel state information and the conjugate transpose matrix; The estimated signal is obtained by multiplying the estimated coefficients and the total received signal.

8. The semantic communication image processing method according to claim 6, characterized in that, When the semantic reasoning model is a neural network model, before inputting the estimated signal into the pre-trained semantic reasoning model corresponding to the transmitting end for semantic reasoning, the method further includes: A training dataset is obtained, which includes a total sample signal and image processing labels. The total sample signal is obtained by the transmitted sample signal generated by each of the transmitting ends based on the sample image. The estimated sample signal corresponding to each of the transmitting ends is obtained based on the total sample signal; The semantic reasoning model is jointly trained using the estimated sample signals to obtain image prediction results; A first loss value is calculated based on the image processing label and the image prediction result; a second loss value is calculated based on the transmitted sample signal and the total sample signal; and a third loss value is calculated based on the transmitted sample signal and the sample image. The total loss value is obtained based on the first loss value, the second loss value, and the third loss value. The semantic reasoning model is then weighted based on the total loss value to obtain the trained semantic reasoning model corresponding to each of the transmitting ends.

9. The semantic communication image processing method according to claim 8, characterized in that, When the semantic reasoning model is a decoder unit, before inputting the estimated signal into the pre-trained semantic reasoning model corresponding to the transmitter for semantic reasoning, the method further includes: A training dataset is obtained, which includes a total sample signal and image processing labels. The total sample signal is obtained by the transmitted sample signal generated by each of the transmitting ends based on the sample image. The estimated sample signal corresponding to each of the transmitting ends is obtained based on the total sample signal; The semantic reasoning model is jointly trained using the estimated sample signals to obtain image prediction results; The total loss value is calculated based on the image processing label and the image prediction result; The semantic reasoning model is weighted based on the total loss value to obtain a trained semantic reasoning model corresponding to each of the transmitting ends.

10. A semantic communication image processing system, characterized in that, include: Multiple transmitting ends, wherein the transmitting ends are used to obtain corresponding transmission signals according to any one of claims 1 to 5; A base station, wherein the base station is configured to acquire the total received signal obtained based on the transmitted signal, and to obtain target image processing data corresponding to each of the transmitting ends according to the semantic communication image processing method according to any one of claims 6 to 9.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the semantic communication image processing method according to any one of claims 1 to 9.

12. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the semantic communication image processing method according to any one of claims 1 to 9.

Images