Semantic communication method and device, network equipment, storage medium and program product

Abstract

The invention discloses a semantic communication method and device, network equipment, a storage medium and a program product, and the method comprises the steps: carrying out the style information coding and content information coding of received original information, and obtaining semantic style information and semantic content information; the original information comprises image information and / or video information; and sending the semantic style information and the semantic content information to the second network equipment. The method further comprises the following steps: receiving semantic style information and semantic content information sent by the first network equipment; decoding the semantic style information to obtain semantic style features, and decoding the semantic content information to obtain semantic content features; and performing information reconstruction according to the semantic content features and the semantic style features to obtain original information. According to the scheme, the style information and the content information are coded, transmitted and decoded respectively, semantic transmission and reconstruction of the original information can be realized based on the essential characteristics of the image, and the semantic communication efficiency and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a semantic communication method, apparatus, network equipment, storage medium and program product. Background Art

[0002] Semantic communication refers to the selective feature extraction, compression and transmission of the original signal, and then the use of semantic level information for communication. The advantage of this technology is that it is task-oriented, understanding first and then transmitting, which greatly improves the transmission efficiency and reliability of the communication system. Semantic communication makes it possible to introduce semantic information in image transmission, which can not only efficiently extract key information in the image, but also save bandwidth.

[0003] Compared with traditional communication schemes, semantic communication systems can effectively improve the reconstruction effect of information images in low signal-to-noise ratio channel scenarios and avoid the "cliff effect" in communication. However, existing semantic communication technologies use source-channel joint coding schemes in image communication to understand and transmit image information, but do not consider the content and style information of the image, resulting in the inability to fully understand and transmit the image. Summary of the invention

[0004] At least one embodiment of the present application provides a semantic communication method, apparatus, network device, storage medium and program product, which are used to solve the problem that in the prior art, only the source-channel joint coding scheme is considered in image communication, but the content information and style information of the image are not considered, and the image cannot be fully and accurately understood and transmitted.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a semantic communication method, which is applied to a first network device, including:

[0007] Performing style information encoding and content information encoding on the received original information respectively to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to characterize the visual features of the original information;

[0008] The semantic style information and the semantic content information are sent to a second network device.

[0009] Furthermore, the received original information is encoded with style features to obtain the semantic style information, including:

[0010] Performing style feature encoding on the original information through a source-channel joint encoding module;

[0011] The source-channel joint coding module includes: a first channel number restriction model;

[0012] The first channel number limitation model is used to adjust the data size of the semantic style information.

[0013] Furthermore, the source-channel joint coding module further includes:

[0014] a first image boundary extension model and / or a first downsampling model;

[0015] The first image boundary extension model is used to extract the global semantic style features of the original information; and the input and output data sizes of the first image boundary extension model are the same;

[0016] The first downsampling model is used to compress the global semantic style features.

[0017] Furthermore, the received original information is encoded with content features, including:

[0018] Encoding the original information for content features to obtain semantic content features;

[0019] Channel coding is performed on the semantic content features to obtain the semantic content information.

[0020] Furthermore, the original information is encoded with content features to obtain semantic content features, including:

[0021] Performing content feature encoding on the original information through a source-channel joint encoding module to obtain the semantic content feature;

[0022] The source-channel joint coding module includes: a second channel number restriction model;

[0023] The second channel number limitation model is used to adjust the data size of the semantic content feature.

[0024] Furthermore, the source-channel joint coding module further includes at least one of the following:

[0025] a second image boundary extension model, a second downsampling model, and a first residual model;

[0026] The second graph boundary extension model is used to extract the global semantic content features of the original information; the input and output data sizes of the second graph boundary extension model are the same;

[0027] The second downsampling model is used to compress the global semantic content features;

[0028] The first residual model is used to extract local semantic content features of the original information.

[0029] Further, channel coding is performed on the semantic content feature, including:

[0030] Channel coding the semantic content features by a source-channel joint coding module;

[0031] The source-channel joint coding module includes at least one of the following: an adaptive downsampling model, a second residual model, and a first channel conversion model;

[0032] Wherein, the adaptive downsampling model is used to compress the semantic content features;

[0033] The second residual model is used to extract local features of the semantic content features and improve the representation capability of the compressed semantic content features;

[0034] The first channel conversion model is used to encode the local features and the compressed semantic content features to obtain the semantic content information, and adjust the data size of the semantic content information.

[0035] In a second aspect, an embodiment of the present application provides a semantic communication method, which is applied to a second network device, including:

[0036] Receiving semantic style information and semantic content information sent by the first network device;

[0037] Decoding the semantic style information to obtain semantic style features, and decoding the semantic content information to obtain semantic content features;

[0038] Information is reconstructed according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

[0039] Furthermore, decoding the semantic style information includes:

[0040] Decoding the semantic style information through a source-channel joint decoding module;

[0041] The source-channel joint decoding module includes: a multi-layer perceptron model;

[0042] The multi-layer perceptron model is used to parse the semantic style information to obtain semantic style features.

[0043] Furthermore, the source-channel joint decoding module further includes:

[0044] An adaptive instance normalization model and / or an adaptive instance normalization channel number limitation model;

[0045] The adaptive instance normalization model is used to dynamically adjust the semantic style features according to the semantic style information;

[0046] The adaptive instance normalization channel number limitation model is used to adjust the data size of the semantic style feature.

[0047] Furthermore, decoding the semantic content information includes:

[0048] Decoding the semantic content information through a source-channel joint decoding module;

[0049] The source-channel joint decoding module includes at least one of the following: a second channel conversion model, a third residual model, and an adaptive upsampling model;

[0050] The second channel conversion model is used to decode the semantic content information to obtain a global semantic content feature; and adjust the data size of the global semantic content feature;

[0051] The third residual model is used to decode the semantic content information to obtain local semantic content features;

[0052] capturing local semantic content features of the semantic content information;

[0053] The adaptive upsampling model is used to adjust the data size of the semantic content feature determined according to the global semantic content feature and the local semantic content feature.

[0054] Further, information reconstruction is performed according to the semantic content feature and the semantic style feature to obtain the original information, including:

[0055] Reconstructing information of the semantic content feature and the semantic style feature through a source-channel joint decoding module;

[0056] The source-channel joint decoding module includes: a content-style adaptive decision model;

[0057] The content style adaptive decision model determines the weights of the semantic style features and the semantic content features in the information reconstruction process according to the user's quality requirements for the original information and the channel environment.

[0058] Furthermore, the source-channel joint decoding module further includes:

[0059] Adaptive instance normalization residual models and / or upsampling models;

[0060] The adaptive instance normalized residual model is used to match the semantic content feature with the semantic style feature to obtain the original information;

[0061] The upsampling model is used to adjust the dimension or data size of the original information.

[0062] In a third aspect, an embodiment of the present application provides a semantic communication device, applied to a first network device, including:

[0063] An encoding module, used to encode the style information and the content information of the received original information respectively, to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to characterize the visual features of the original information;

[0064] The sending module is used to send the semantic style information and the semantic content information to the second network device.

[0065] In a fourth aspect, an embodiment of the present application provides a semantic communication device, applied to a second network device, including:

[0066] A receiving module, configured to receive semantic style information and semantic content information sent by the first network device;

[0067] A decoding module, used to decode the semantic style information to obtain semantic style features, and to decode the semantic content information to obtain semantic content features;

[0068] The reconstruction module is used to reconstruct information according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

[0069] In a fifth aspect, an embodiment of the present application provides a network device, including a transceiver and a processor, wherein:

[0070] The processor is used to encode the style information and the content information of the received original information respectively to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to characterize the visual features of the original information;

[0071] The transceiver is used to send the semantic style information and the semantic content information to the second network device.

[0072] In a sixth aspect, an embodiment of the present application provides a network device, including a transceiver and a processor, wherein:

[0073] The transceiver is used to receive the semantic style information and semantic content information sent by the first network device;

[0074] The processor is used to decode the semantic style information to obtain semantic style features, and to decode the semantic content information to obtain semantic content features;

[0075] Information is reconstructed according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

[0076] In the seventh aspect, an embodiment of the present application provides a network device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the first aspect or the second aspect.

[0077] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0078] In a ninth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the second aspect.

[0079] Compared with the prior art, the semantic communication method, apparatus, network device, storage medium and program product provided in the embodiments of the present application can encode the style information and content information of the original information including image information and / or video information respectively, and then send the encoded semantic style information and the semantic content information to the second network device. The scheme of the present invention can realize the semantic transmission and reconstruction of the original information based on the essential characteristics of the image by encoding and transmitting the style information and the content information respectively, thereby improving the efficiency and quality of semantic communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0081] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0082] Figure 2 A flowchart of a semantic communication method in the prior art;

[0083] Figure 3 This is a flow chart of the semantic communication method according to an embodiment of the present application when it is applied to a first network device;

[0084] Figure 4 This is a flow chart of the semantic communication method according to an embodiment of the present application when it is applied to a second network device;

[0085] Figure 5 This is a flow chart of a semantic communication method according to another embodiment of the present application when it is applied to a second network device;

[0086] Figure 6 This is a flow chart of a semantic communication method according to another embodiment of the present application when it is applied to a second network device;

[0087] Figure 7 This is a structural diagram of a semantic communication device when applied to a first network device according to an embodiment of the present application;

[0088] Figure 8 A schematic diagram of the structure of a semantic communication device when applied to a second network device according to an embodiment of the present application;

[0089] Fig. 9 A schematic diagram of the structure of a network device according to an embodiment of the present application;

[0090] Fig.10 A schematic diagram of the structure of a network device according to another embodiment of the present application;

[0091] Fig.11 This is a schematic diagram of the structure of a network device according to another embodiment of the present application. DETAILED DESCRIPTION

[0092] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0093] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. "And / or" in the specification and claims represents at least one of the connected objects.

[0094] The technology described herein is not limited to NR systems and Long Time Evolution (LTE) / LTE-Advanced (LTE-A) systems, and can also be used for various wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) and other systems. The terms "system" and "network" are often used interchangeably. A CDMA system can implement radio technologies such as CDMA2000, Universal Terrestrial Radio Access (UTRA). UTRA includes Wideband Code Division Multiple Access (WCDMA) and other CDMA variants. A TDMA system can implement radio technologies such as Global System for Mobile Communication (GSM). OFDMA systems can implement radio technologies such as Ultra Mobile Broadband (UMB), Evolution-UTRA (E-UTRA), IEEE 802.21 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, Flash-OFDM, etc. UTRA and E-UTRA are parts of Universal Mobile Telecommunications System (UMTS). LTE and more advanced LTE (such as LTE-A) are new UMTS versions that use E-UTRA. UTRA, E-UTRA, UMTS, LTE, LTE-A, and GSM are described in documents from an organization named "3rd Generation Partnership Project" (3GPP). CDMA2000 and UMB are described in documents from an organization named "3rd Generation Partnership Project 2" (3GPP2).The techniques described herein may be used for the systems and radio technologies mentioned above as well as for other systems and radio technologies. However, the following description describes an NR system for example purposes and uses NR terminology in much of the following description, although the techniques may also be applicable to applications other than NR system applications.

[0095] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the functions and arrangements of the elements discussed without departing from the spirit and scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0096] See also Figure 1 , Figure 1 A block diagram of a wireless communication system applicable to an embodiment of the present application is shown. The wireless communication system includes a terminal 11 and a network device 12. The terminal 11 may also be referred to as a user terminal or user equipment (UE, User Equipment), and the terminal 11 may be a terminal side device such as a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a personal digital assistant (Personal Digital Assistant, PDA), a mobile Internet device (Mobile Internet Device, MID), a wearable device (Wearable Device) or a vehicle-mounted device. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network device 12 can be a base station and / or a core network element, wherein the above-mentioned base station can be a base station of 5G and later versions (for example: gNB, 5G NR NB, etc.), or a base station in other communication systems (for example: eNB, WLAN access point, or other access point, etc.), wherein the base station can be called a node B, an evolved node B, an access point, a base transceiver station (Base Transceiver Station, BTS), a radio base station, a radio transceiver, a basic service set (Basic Service Set, BSS), an extended service set (Extended Service Set, ESS), a B node, an evolved B node (eNB), a home B node, a home evolved B node, a WLAN access point, a WiFi node or some other suitable term in the field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is taken as an example, but the specific type of the base station is not limited.

[0097] The base station can communicate with the terminal 11 under the control of a base station controller, and in various examples, the base station controller can be part of the core network or certain base stations. Some base stations can communicate control information or user data with the core network via a backhaul. In some examples, some of these base stations can communicate with each other directly or indirectly via a backhaul link, which can be a wired or wireless communication link. The wireless communication system can support operations on multiple carriers (waveform signals of different frequencies). A multi-carrier transmitter can transmit modulated signals on these multiple carriers at the same time. For example, each communication link can be a multi-carrier signal modulated according to various radio technologies. Each modulated signal can be sent on a different carrier and can carry control information (e.g., reference signals, control channels, etc.), overhead information, data, etc.

[0098] The base station can communicate wirelessly with the terminal 11 via one or more access point antennas. Each base station can provide communication coverage for its respective coverage area. The coverage area of ​​the access point can be divided into sectors that constitute only a portion of the coverage area. The wireless communication system may include different types of base stations (e.g., macro base stations, micro base stations, or micro-micro base stations). The base station may also utilize different radio technologies, such as cellular or WLAN radio access technologies. The base station may be associated with the same or different access networks or operator deployments. The coverage areas of different base stations (including coverage areas of the same or different types of base stations, coverage areas utilizing the same or different radio technologies, or coverage areas belonging to the same or different access networks) may overlap.

[0099] The communication link in the wireless communication system may include an uplink for carrying an uplink (UL) transmission (e.g., from the terminal 11 to the network device 12), or a downlink for carrying a downlink (DL) transmission (e.g., from the network device 12 to the terminal 11). UL transmission may also be referred to as reverse link transmission, and DL transmission may also be referred to as forward link transmission. Downlink transmission may be performed using a licensed frequency band, an unlicensed frequency band, or both. Similarly, uplink transmission may be performed using a licensed frequency band, an unlicensed frequency band, or both.

[0100] like Figure 2 As shown, in the prior art, the semantic communication transmitter uses a semantic source-channel joint coding module to semantically encode the source image, but the semantic coding is limited to the content information of the source image;

[0101] After semantic coding, channel coding is performed, and then the encoded source image enters the channel;

[0102] The receiving end receives the encoded source image, first performs channel decoding through a source-channel joint decoding module, and then performs semantic decoding to restore the source image.

[0103] like Figure 3 As shown, a semantic communication method provided in an embodiment of the present application, when applied to a first network device, includes the following steps:

[0104] Step 301, respectively encoding style information and content information of received original information to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to characterize visual features of the original information.

[0105] Optionally, the first network device is a sending end device.

[0106] Optionally, the semantic content information is used to represent detailed information of an image or video;

[0107] The semantic style information is used to characterize the overall visual features of an image or video.

[0108] It should be noted that the semantic content information can be used for tasks such as object recognition and image retrieval;

[0109] The semantic style information can be used in application scenarios such as style transfer and art generation.

[0110] In the embodiment of the present invention, the semantic content information includes specific objects in the image or video; for example, objects, text, scenes, etc. in the image or video;

[0111] The semantic style information includes the color, texture or composition of the image or video.

[0112] Step 302: Send the semantic style information and the semantic content information to a second network device.

[0113] The semantic communication method provided in the embodiment of the present application can encode the style information and content information of the original information including the image information and / or the video information respectively, and then send the encoded semantic style information and the semantic content information to the second network device. The scheme of the present invention can combine the semantic content information and the semantic style information by encoding the style information, realize the semantic transmission and reconstruction of the original information based on the essential characteristics of the image, and improve the efficiency and quality of semantic communication.

[0114] Optionally, encoding the received original information with style features to obtain the semantic style information includes:

[0115] Performing style feature encoding on the original information through a source-channel joint encoding module;

[0116] The source-channel joint coding module includes: a first channel number restriction model;

[0117] The first channel number limitation model is used to adjust the data size of the semantic style information.

[0118] Optionally, the source-channel joint coding module is deployed in the first network terminal.

[0119] The first channel number limitation model is used to adjust the data size of the semantic style information, which can be understood as:

[0120] By setting the number of output channels of the first channel number restriction model, the data size of the semantic style information output by the source-channel joint encoding module can be adjusted.

[0121] In an embodiment of the present invention, the first channel number limitation model includes: an adaptive average pooling layer and a convolution layer with a 1x1 convolution kernel, a step size of 1, and an output channel number of 8.

[0122] It should be noted that the adaptive average pooling layer is used to reduce the spatial dimension of the original information, and then further extract and compress the semantic style information through the convolution kernel and the output channel; by adjusting the number of channels of the first channel number limitation model, the size of the image semantic style information can be freely selected, thereby achieving precise control of the image style information.

[0123] Optionally, the source-channel joint coding module further includes:

[0124] a first image boundary extension model and / or a first downsampling model;

[0125] The first image boundary extension model is used to extract the global semantic style features of the original information; and the input and output data sizes of the first image boundary extension model are the same;

[0126] The first downsampling model is used to compress the global semantic style features.

[0127] Optionally, the first image boundary extension model, the first downsampling model and the first channel number limitation model are connected in sequence to constitute a part of the source-channel joint coding model.

[0128] In the embodiment of the present invention, the first image boundary extension model includes: a convolution layer with a 7x7 convolution kernel, a step size of 1, and a padding of 3;

[0129] Furthermore, an activation function layer is provided before the convolution layer to activate the nonlinear representation of features, thereby extracting global features of the image.

[0130] In an embodiment of the present invention, the first downsampling model is a convolutional layer structure consisting of a 4x4 convolution kernel, a step size of 2, and a padding of 1; by setting the step size to 2, the data or dimension of the global feature can be halved.

[0131] Furthermore, an activation function layer is set before each convolutional layer.

[0132] The semantic communication method of the embodiment of the present invention can keep the size of the output data the same as the size of the input data through the first image boundary extension model, and at the same time prevent the convolution operation from introducing unnecessary boundary effects at the edge of the image; the first downsampling model can effectively reduce the spatial dimension of the global semantic style feature, thereby capturing important features in the input image, and the convolution operation can control the data size of the global semantic style feature through the setting of the step size.

[0133] Optionally, content feature encoding is performed on the received original information, including:

[0134] Encoding the original information for content features to obtain semantic content features;

[0135] Channel coding is performed on the semantic content features to obtain the semantic content information.

[0136] It should be noted that the data of the semantic content information is much larger than the data of the semantic style information, and the semantic content information needs to be further encoded to reduce the data size of the semantic content information entering the channel, thereby improving transmission efficiency.

[0137] Optionally, encoding the original information for content features to obtain semantic content features includes:

[0138] Performing content feature encoding on the original information through a source-channel joint encoding module to obtain the semantic content feature;

[0139] The source-channel joint coding module includes: a second channel number restriction model;

[0140] The second channel number limitation model is used to adjust the data size of the semantic content feature.

[0141] Optionally, the second channel number limitation model is used to adjust the data size of the semantic content feature, which can be understood as:

[0142] By setting the number of output channels of the second channel number restriction model, the data size of the semantic content features output by the source channel joint encoding module can be adjusted.

[0143] In the embodiment of the present invention, the second channel number restriction model includes: a convolution layer with a 1x1 convolution kernel, a step size of 1, and an output channel number of 128;

[0144] Furthermore, an instance normalization layer and an activation function layer are provided before the convolution layer.

[0145] The semantic communication method of the embodiment of the present invention can limit and adjust the number of channels of the output semantic content features through the second channel number limitation model, thereby controlling the dimension and representation ability of the output semantic content features.

[0146] Optionally, the source-channel joint coding module further includes at least one of the following:

[0147] a second image boundary extension model, a second downsampling model, and a first residual model;

[0148] The second graph boundary extension model is used to extract the global semantic content features of the original information; the input and output data sizes of the second graph boundary extension model are the same;

[0149] The second downsampling model is used to compress the global semantic content features;

[0150] The first residual model is used to extract local semantic content features of the original information.

[0151] Optionally, the second image boundary extension model, the second downsampling model, the first residual model and the second channel number restriction model are connected in sequence to constitute a part of the source-channel joint coding model.

[0152] In the embodiment of the present invention, the second image boundary extension model includes: a convolution layer with a 7x7 convolution kernel, a step size of 1, and a padding of 3;

[0153] Furthermore, an instance normalization layer and an activation function layer are set before the convolution layer to activate the nonlinear representation of the features.

[0154] In the embodiment of the present invention, the second downsampling model includes: a convolution layer structure with a 4x4 convolution kernel, a step size of 2, and a padding of 1;

[0155] Furthermore, an instance normalization layer and an activation function layer are set before each convolutional layer.

[0156] It should be noted that the addition of the activation function layer helps to introduce nonlinear transformations, thereby enhancing the representation ability of the model, enabling it to better learn the complex features in the input image and improve the performance and generalization ability of the model.

[0157] In the embodiment of the present invention, the first residual model includes: a convolution layer structure with a 4x4 convolution kernel, a step size of 2, and a padding of 1;

[0158] Furthermore, an instance normalization layer and an activation function layer are set before each convolution layer to capture the local features of the image;

[0159] A cross-layer residual connection is performed between every two convolutional layers to achieve cross-layer transmission of information and improve the representation ability of features.

[0160] The semantic communication method of the embodiment of the present invention can not only extract the local semantic content features of the original information through the first residual model, but also solve the gradient disappearance or gradient explosion problem in the model training process, improve the training speed of the model, and improve the stability and generalization ability of the model.

[0161] Optionally, channel coding the semantic content feature includes:

[0162] Channel coding the semantic content features by a source-channel joint coding module;

[0163] The source-channel joint coding module includes at least one of the following: an adaptive downsampling model, a second residual model, and a first channel conversion model;

[0164] Wherein, the adaptive downsampling model is used to compress the semantic content features;

[0165] The second residual model is used to extract local features of the semantic content features and improve the representation capability of the compressed semantic content features;

[0166] The first channel conversion model is used to encode the local features and the compressed semantic content features to obtain the semantic content information, and adjust the data size of the semantic content information.

[0167] In the embodiment of the present invention, the adaptive downsampling model, the second residual model and the first channel conversion model are connected in sequence to form a part of the source-channel joint coding module.

[0168] In the embodiment of the present invention, the adaptive downsampling model includes: a convolution layer structure with a 4x4 convolution kernel, a step size of 2, and a padding of 1;

[0169] Furthermore, an instance normalization layer and an activation function layer are set before each convolution layer;

[0170] The number of convolutions is determined by the channel conditions to optimize the system computing power consumption and image restoration capabilities in different communication scenarios.

[0171] In the embodiment of the present invention, the second residual model includes: a convolution layer structure with a 3x3 convolution kernel, a step size of 1, and a padding of 1;

[0172] Furthermore, an instance normalization layer and an activation function layer are set before each convolution layer to capture the local features of the image; and a cross-layer residual connection is performed between every two convolution layers to form a residual block to achieve cross-layer transmission of information and improve the representation capability of the semantic content features.

[0173] In the embodiment of the present invention, the first channel conversion model includes: a convolution layer with a 3x3 convolution kernel, a step size of 1, a padding of 1, and an output channel number of C;

[0174] Furthermore, an instance normalization layer and an activation function layer are set before the convolution layer.

[0175] The semantic communication method of the embodiment of the present invention can compress the semantic content features through the adaptive downsampling model, reduce the size of the transmitted data, and improve the transmission rate; through the second residual model, the representation ability of the semantic content features can be improved; through the first channel conversion model, the number of channels of the semantic content information obtained through the channel coding module can be adaptively limited and adjusted, thereby controlling the dimension or size of the content information entering the channel, thereby controlling the system bandwidth occupancy and improving the transmission rate.

[0176] like Figure 4 As shown, the embodiment of the present invention further provides a semantic communication method, which, when applied to a second network device, comprises the following steps:

[0177] Step 401, receiving semantic style information and semantic content information sent by a first network device;

[0178] Step 402, decoding the semantic style information to obtain semantic style features, and decoding the semantic content information to obtain semantic content features;

[0179] Step 403: reconstruct information according to the semantic content feature and the semantic style feature to obtain original information; the original information includes image information and / or video information.

[0180] The semantic communication method provided in the embodiment of the present application decodes the semantic style information and the semantic content information obtained after respectively encoding the style information and the content information of the original information, and then reconstructs the semantic style information and the semantic content information to restore the original information. The scheme of the present invention combines the encoding and decoding of the semantic style information with the encoding and decoding of the semantic content information, thereby improving the transmission speed of the original information while ensuring the degree of information retention in the encoding and decoding process, thereby improving the quality and dimension of the original information returned.

[0181] Optionally, decoding the semantic style information includes:

[0182] Decoding the semantic style information through a source-channel joint decoding module;

[0183] The source-channel joint decoding module includes: a multi-layer perceptron model;

[0184] The multi-layer perceptron model is used to parse the semantic style information to obtain semantic style features.

[0185] Optionally, the multilayer perceptron model includes: three linear layers, and an activation function layer is respectively provided after the first two linear layers.

[0186] In the embodiment of the present invention, the dimension of the input vector of the first linear layer of the multilayer perceptron model is 8, and the dimension of the output vector is 256; the dimension of the input vector of the second linear layer is 256, and the dimension of the output vector is 256; the dimension of the input vector of the third linear layer is 256, and the dimension of the output vector is 4608;

[0187] The activation function layer after the first two linear layers uses the ReLu activation function.

[0188] Specifically, the multi-layer perceptron model takes the semantic style information received through the channel as input and maps it into parameters, which will serve as the number of input parameters of the adaptive instance normalization model.

[0189] Optionally, the source-channel joint decoding module further includes:

[0190] An adaptive instance normalization model and / or an adaptive instance normalization channel number limitation model;

[0191] Wherein, the adaptive instance normalization model is used to dynamically adjust the semantic style features according to the semantic style information;

[0192] The adaptive instance normalization channel number limitation model is used to adjust the data size of the semantic style feature.

[0193] Optionally, the multilayer perceptron model, the adaptive instance normalization model and the adaptive instance normalization channel number limitation model are connected in sequence to constitute a part of the source-channel joint decoding module.

[0194] In the embodiment of the present invention, the adaptive instance normalization model includes: an instance normalization layer and two linear layers, which are used to learn the scaling factor and offset factor of the style vector.

[0195] Specifically, the adaptive instance normalization model dynamically adjusts the normalization of the semantic style information of the received image to obtain the mean and variance of the semantic style information, thereby realizing the style representation of the original information.

[0196] Optionally, the adaptive instance normalization channel number limitation model includes: a deconvolution layer with a 3x3 convolution kernel, a step size of 1, a padding of 1, and an input channel number of 128;

[0197] Furthermore, an adaptive instance normalization layer and an activation function layer are set before the convolution layer;

[0198] The adaptive instance normalization channel number limitation model of the embodiment of the present invention can adjust the number of channels of semantic style information to correspond to the number of channels of semantic style information extracted in the semantic encoding module, thereby improving the stability and performance of the system.

[0199] Optionally, decoding the semantic content information includes:

[0200] Decoding the semantic content information through a source-channel joint decoding module;

[0201] The source-channel joint decoding module includes at least one of the following: a second channel conversion model, a third residual model, and an adaptive upsampling model;

[0202] The second channel conversion model is used to decode the semantic content information to obtain a global semantic content feature; and adjust the data size of the global semantic content feature;

[0203] The third residual model is used to decode the semantic content information to obtain local semantic content features;

[0204] capturing local semantic content features of the semantic content information;

[0205] The adaptive upsampling model is used to adjust the data size of the semantic content feature determined according to the global semantic content feature and the local semantic content feature.

[0206] Optionally, the second channel conversion model includes: a deconvolution layer with a 3x3 convolution kernel, a step size of 1, a padding of 1, and an input channel number of C;

[0207] Furthermore, an instance normalization layer and an activation function layer are set before the deconvolution layer.

[0208] The second channel conversion model of the embodiment of the present invention can adjust the number of channels of the semantic content information entering the decoder to correspond to the number of output channels of the channel coding module, thereby improving the stability and performance of the system.

[0209] Optionally, the third residual model includes: a convolution layer with a 3x3 convolution kernel, a step size of 1, and a padding of 1;

[0210] Furthermore, an instance normalization layer and an activation function layer are provided before each convolution layer to capture the local features of the given content information;

[0211] A cross-layer residual connection is performed between every two convolutional layers to achieve cross-layer transmission of information and improve the representation capability of the local features.

[0212] Optionally, the adaptive upsampling model includes: a deconvolution layer with a 4x4 convolution kernel, a step size of 2, and a padding of 1;

[0213] Specifically, an instance normalization layer and an activation function layer are set before each deconvolution layer;

[0214] The number of deconvolution layers is determined by the channel quality and is the same as the number of convolution layers in the adaptive downsampling model of the source-channel joint coding module.

[0215] The adaptive upsampling model of the embodiment of the present invention can upsample the input semantic content information through a deconvolution operation, thereby restoring the size and details of the original information, expanding the spatial dimension of the semantic content, and providing support for subsequent reconstruction and generation of the original information.

[0216] Optionally, reconstructing information according to the semantic content feature and the semantic style feature to obtain the original information includes:

[0217] Reconstructing information of the semantic content feature and the semantic style feature through a source-channel joint decoding module;

[0218] The source-channel joint decoding module includes: a content-style adaptive decision model;

[0219] The content style adaptive decision model determines the weights of the semantic style features and the semantic content features in the information reconstruction process according to the user's quality requirements for the original information and the channel environment.

[0220] Optionally, the core of the adaptive mechanism of the content-style adaptive decision model is to dynamically adjust the ratio of semantic content information to semantic style information according to the user's requirements for restoration quality and the current channel environment.

[0221] Optionally, the channel environment may be detected by the source-channel joint decoding module, or may be detected by other modules and then sent to the source-channel joint decoding module.

[0222] In the embodiment of the present invention, the quality of the restored original information can be adjusted by setting parameters according to the user's requirements for the restoration quality of the original information; the restoration quality requirements may include image clarity, artistic effect, etc.;

[0223] The channel environment is evaluated by detecting parameters such as the bandwidth, delay, and bit error rate of the current channel to ensure the stability and capacity of channel transmission.

[0224] Optionally, determining the weights of the semantic style features and the semantic content features in the information reconstruction process includes:

[0225] According to the user's quality requirements for the original information and the channel environment, the weights of the semantic style feature and the semantic content feature are calculated by the following formula:

[0226]

[0227] Among them, R is the original information after information reconstruction, C and S represent semantic content information and semantic style information respectively, and α and β are weight coefficients.

[0228] The content style adaptive decision model of the embodiment of the present invention can flexibly adjust the dimension and size of the original information after information reconstruction according to user needs.

[0229] Optionally, the source-channel joint decoding module further includes:

[0230] Adaptive instance normalization residual models and / or upsampling models;

[0231] The adaptive instance normalized residual model is used to match the semantic content feature with the semantic style feature to obtain the original information;

[0232] The upsampling model is used to adjust the dimension or data size of the original information.

[0233] Optionally, the adaptive instance normalized residual model includes: a convolution layer structure with a 3x3 convolution kernel, a step size of 1, and a padding of 1;

[0234] Furthermore, an adaptive instance normalization layer and an activation function layer are set before each convolution layer.

[0235] The adaptive instance normalization residual model of the embodiment of the present invention takes the semantic style information transmitted by the channel as input. The adaptive instance normalization model dynamically adjusts the mean and variance of the input semantic style information so that the statistical characteristics of the semantic style information match the semantic style information of the original information, thereby achieving style correspondence; at the same time, a cross-layer residual connection is performed between every two convolutional layers to achieve cross-layer transmission of information and improve the feature representation capability.

[0236] Optionally, the upsampling model includes: a deconvolution layer structure with a 4x4 convolution kernel, a step size of 2, and a padding of 1;

[0237] Specifically, a layer normalization layer and an activation function layer are set before each deconvolution layer.

[0238] The upsampling model of the embodiment of the present invention can reduce internal covariate migration by setting a normalization layer, reduce the distribution change of the input of each layer in the neural network, improve the generalization ability of the model, and enhance the robustness of the model to the input.

[0239] It should be noted that the first network device and the second network device need to collaborate and update the semantic knowledge base;

[0240] The semantic knowledge base has a semantic information processing model for processing multiple modal information and has a hierarchical structure; the transceiver device (the first network device and the second network device) can update the content of the semantic knowledge base according to the newly acquired information or the semantic information processing model; the knowledge base of the transceiver device realizes knowledge sharing and synchronization through collaborative updating, so that the entire system can continuously obtain new semantic knowledge from each endpoint and integrate it into its own semantic processing model, keeping the knowledge base of the entire system always up to date, thereby continuously prompting the intelligence level and application capabilities of the system to improve the semantic understanding and application capabilities of the entire system.

[0241] In the embodiment of the present invention, the transmitting and receiving ends are respectively provided with local private knowledge bases with information privacy protection measures, storing private semantic knowledge shared by themselves or only with designated users; at the same time, the establishment and update of the semantic knowledge base can be completed by using the generalization ability and flexible structure of deep neural networks, machine learning methods and communication protocols to ensure reliable transmission and synchronous update of information. For example, the consistency protocol in the distributed system can be used to ensure data consistency between different devices, and the incremental update strategy can be used to reduce the overhead of data transmission.

[0242] In the embodiment of the present invention, the decoding of the semantic style information and the information reconstruction of the semantic content features and the semantic style features are concentrated in one model.

[0243] like Figure 5 and Figure 6 As shown, the source-channel joint coding module in the first network device includes two parts, namely, style information coding and content information coding;

[0244] At the same time, for the content information, since its data is large, after channel coding (i.e., semantic information downsampling), the style information and content information enter the channel;

[0245] The source channel joint decoding module in the second network device includes two parts, namely style information decoding and content information decoding;

[0246] Then, the semantic style information and semantic content information obtained by decoding are reconstructed to obtain the original information.

[0247] The above describes various methods of the embodiments of the present application. The following further provides devices for implementing the above methods.

[0248] like Figure 7 As shown, the embodiment of the present invention further provides a semantic communication device 700, which is applied to a first network device, including:

[0249] The encoding module 701 is used to encode the style information and the content information of the received original information to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to represent the visual features of the original information;

[0250] The sending module 702 is configured to send the semantic style information and the semantic content information to a second network device.

[0251] The semantic communication device provided in the embodiment of the present application can encode the style information and content information of the original information including the image information and / or the video information respectively, and then send the encoded semantic style information and the semantic content information to the second network device. The scheme of the present invention can combine the semantic content information and the semantic style information by encoding the style information, realize the semantic transmission and reconstruction of the original information based on the essential characteristics of the image, and improve the efficiency and quality of semantic communication.

[0252] like Figure 8 As shown, the embodiment of the present invention further provides a semantic communication device 800, which is applied to a second network device, including:

[0253] The receiving module 801 is used to receive the semantic style information and the semantic content information sent by the first network device;

[0254] A decoding module 802, configured to decode the semantic style information to obtain semantic style features, and to decode the semantic content information to obtain semantic content features;

[0255] The reconstruction module 803 is used to reconstruct information according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

[0256] The semantic communication device provided in the embodiment of the present application decodes the semantic style information and the semantic content information obtained after respectively encoding the style information and the content information of the original information, and then reconstructs the semantic style information and the semantic content information to restore the original information. The solution of the present invention combines the encoding and decoding of the semantic style information with the encoding and decoding of the semantic content information, thereby improving the transmission speed of the original information while ensuring the degree of information retention in the encoding and decoding process, thereby improving the quality and dimension of the original information returned.

[0257] like Fig. 9 As shown, the embodiment of the present application further provides a network device 900, including: a transceiver 901 and a processor 902;

[0258] The processor 902 is used to encode the style information and the content information of the received original information respectively to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to represent the visual features of the original information;

[0259] The transceiver 901 is used to send the semantic style information and the semantic content information to the second network device.

[0260] Please refer to Fig.10 , the embodiment of the present application further provides a network device 1000, including: a transceiver 1001 and a processor 1002;

[0261] The transceiver 1001 is used to receive the semantic style information and semantic content information sent by the first network device;

[0262] The processor 1002 is configured to decode the semantic style information to obtain a semantic style feature, and decode the semantic content information to obtain a semantic content feature;

[0263] Information is reconstructed according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

[0264] Please refer to Fig.11 The embodiment of the present application also provides a network device 1100, including a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the computer program is executed by the processor 1101, the computer program implements the various processes of the above-mentioned semantic communication method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0265] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned semantic communication method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0266] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned semantic communication method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0267] It should be noted that, in this article, the terms "include", "include" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0268] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0269] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A semantic communication method, applied to a first network device, characterized in that: include: The received original information is respectively encoded with style information and content information to obtain semantic style information and semantic content information; The original information includes image information and / or video information; The semantic style information is used to represent the visual features of the original information; The semantic style information and the semantic content information are sent to a second network device.

2. The semantic communication method according to claim 1, characterized in that: The received original information is encoded with style features to obtain the semantic style information, including: Performing style feature encoding on the original information through a source-channel joint encoding module; The source-channel joint coding module includes: a first channel number restriction model; The first channel number limitation model is used to adjust the data size of the semantic style information.

3. The semantic communication method according to claim 2, characterized in that: The source-channel joint coding module further includes: a first image boundary extension model and / or a first downsampling model; The first image boundary extension model is used to extract the global semantic style features of the original information; and the input and output data sizes of the first image boundary extension model are the same; The first downsampling model is used to compress the global semantic style features.

4. The semantic communication method according to claim 1, characterized in that: Encode the content characteristics of the received original information, including: Encoding the original information for content features to obtain semantic content features; Channel coding is performed on the semantic content features to obtain the semantic content information.

5. The semantic communication method according to claim 4, characterized in that: The original information is encoded with content features to obtain semantic content features, including: Performing content feature encoding on the original information through a source-channel joint encoding module to obtain the semantic content feature; The source-channel joint coding module includes: a second channel number restriction model; The second channel number limitation model is used to adjust the data size of the semantic content feature.

6. The semantic communication method according to claim 5, characterized in that: The source-channel joint coding module further includes at least one of the following: a second image boundary extension model, a second downsampling model, and a first residual model; The second graph boundary extension model is used to extract the global semantic content features of the original information; the input and output data sizes of the second graph boundary extension model are the same; The second downsampling model is used to compress the global semantic content features; The first residual model is used to extract local semantic content features of the original information.

7. The semantic communication method according to claim 4, characterized in that: Channel coding the semantic content feature includes: Channel coding the semantic content features by a source-channel joint coding module; The source-channel joint coding module includes at least one of the following: an adaptive downsampling model, a second residual model, and a first channel conversion model; Wherein, the adaptive downsampling model is used to compress the semantic content features; The second residual model is used to extract local features of the semantic content features and improve the representation capability of the compressed semantic content features; The first channel conversion model is used to encode the local features and the compressed semantic content features to obtain the semantic content information, and adjust the data size of the semantic content information.

8. A semantic communication method, applied to a second network device, characterized in that: include: Receiving semantic style information and semantic content information sent by the first network device; Decoding the semantic style information to obtain semantic style features, and decoding the semantic content information to obtain semantic content features; Information is reconstructed according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

9. The semantic communication method according to claim 8, characterized in that: Decoding the semantic style information includes: Decoding the semantic style information through a source-channel joint decoding module; The source-channel joint decoding module includes: a multi-layer perceptron model; The multi-layer perceptron model is used to parse the semantic style information to obtain semantic style features.

10. The semantic communication method according to claim 9, characterized in that: The source-channel joint decoding module further includes: An adaptive instance normalization model and / or an adaptive instance normalization channel number limitation model; Wherein, the adaptive instance normalization model is used to dynamically adjust the semantic style features according to the semantic style information; The adaptive instance normalization channel number limitation model is used to adjust the data size of the semantic style feature.

11. The semantic communication method according to claim 8, characterized in that: Decoding the semantic content information includes: Decoding the semantic content information through a source-channel joint decoding module; The source-channel joint decoding module includes at least one of the following: a second channel conversion model, a third residual model, and an adaptive upsampling model; The second channel conversion model is used to decode the semantic content information to obtain a global semantic content feature; and adjust the data size of the global semantic content feature; The third residual model is used to decode the semantic content information to obtain local semantic content features; capturing local semantic content features of the semantic content information; The adaptive upsampling model is used to adjust the data size of the semantic content feature determined according to the global semantic content feature and the local semantic content feature.

12. The semantic communication method according to claim 8, characterized in that: Reconstructing information according to the semantic content feature and the semantic style feature to obtain the original information includes: Reconstructing information of the semantic content feature and the semantic style feature through a source-channel joint decoding module; The source-channel joint decoding module includes: a content-style adaptive decision model; The content style adaptive decision model determines the weights of the semantic style features and the semantic content features in the information reconstruction process according to the user's quality requirements for the original information and the channel environment.

13. The semantic communication method according to claim 12, characterized in that: The source-channel joint decoding module further includes: Adaptive instance normalization residual models and / or upsampling models; The adaptive instance normalized residual model is used to match the semantic content feature with the semantic style feature to obtain the original information; The upsampling model is used to adjust the dimension or data size of the original information.

14. A semantic communication device, applied to a first network device, characterized in that: include: An encoding module, used for encoding the style information and the content information of the received original information respectively, to obtain semantic style information and semantic content information; The original information includes image information and / or video information; The semantic style information is used to represent the visual features of the original information; The sending module is used to send the semantic style information and the semantic content information to the second network device.

15. A semantic communication device, applied to a second network device, characterized in that: include: A receiving module, configured to receive semantic style information and semantic content information sent by the first network device; A decoding module, used to decode the semantic style information to obtain semantic style features, and to decode the semantic content information to obtain semantic content features; The reconstruction module is used to reconstruct information according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

16. A network device, characterized in that: comprising a transceiver and a processor, wherein: The processor is used to encode the style information and the content information of the received original information respectively to obtain semantic style information and semantic content information; the original information includes image information and / or video information; the semantic style information is used to characterize the visual features of the original information; The transceiver is used to send the semantic style information and the semantic content information to the second network device.

17. A network device, characterized in that: comprising a transceiver and a processor, wherein: The transceiver is used to receive the semantic style information and semantic content information sent by the first network device; The processor is used to decode the semantic style information to obtain semantic style features, and to decode the semantic content information to obtain semantic content features; Information is reconstructed according to the semantic content features and the semantic style features to obtain original information; the original information includes image information and / or video information.

18. A network device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7; or, the program, when executed by the processor, implements the steps of the method according to any one of claims 8 to 13.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.

20. A computer program product, characterized in that The method comprises computer instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images