Information processing method and device, equipment, storage medium and computer program product

Abstract

The invention provides an information processing method and device, equipment, a storage medium and a computer program product, and the information processing method comprises the steps: extracting semantic features for training data through a semantic encoder, obtaining a similar codebook corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and storing the similar codebook in the semantic knowledge base; mapping the index of the similar codebook into first digital information for transmission; using a semantic knowledge base to determine a similar codebook according to the received first digital information, and transmitting the similar codebook to a semantic decoder; carrying out feature fusion on the similar codebook by utilizing a semantic decoder, and then carrying out decoding to obtain decoded data corresponding to the training data; obtaining a processing loss value according to the training data, the decoding data, the semantic features and the similar codebook; and according to the processing loss value, updating a model parameter of the semantic encoder, a quantization codebook parameter of the semantic knowledge base and a model parameter of the semantic decoder. According to the scheme, the problem that a semantic communication system is difficult to actually deploy in the prior art can be solved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an information processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Semantic communication is a new type of communication technology that encodes and transmits information by extracting semantic features. It can break through the theoretical limitations of traditional syntactic communication. Its core lies in the receiver restoring the semantics rather than bit stream matching. It realizes a communication paradigm change of "understanding before transmission" through artificial intelligence technology.

[0003] Semantic communication, aided by a shared semantic knowledge base, extracts semantic features from the information source for encoding and transmission, achieving the "meaningful" delivery of information. It is considered one of the enabling technologies for realizing the deep integration of 6G communication and AI (artificial intelligence). As a key technology of semantic communication, Deep Joint Source-Channel Coding (DJSCC) has recently received widespread attention. Semantic communication systems built upon this technology typically use DJSCCC to extract semantic information before transmission to verify the effectiveness of semantic communication under different channel conditions. However, such semantic communication systems usually employ analog communication methods, which are inconsistent with currently used digital communication methods, making effective deployment difficult.

[0004] As can be seen from the above, existing semantic communication systems based on deep source-channel joint coding suffer from problems such as difficulty in practical deployment. Summary of the Invention

[0005] The purpose of this application is to provide an information processing method, apparatus, device, storage medium, and computer program product to solve the problem that semantic communication systems in the prior art are difficult to deploy in practice.

[0006] To address the aforementioned technical problems, embodiments of this application provide an information processing method, including:

[0007] Using a semantic encoder, semantic features are extracted from the training data, and similar codebooks corresponding to the semantic features are obtained from the quantized codebooks stored in the semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission.

[0008] Using a semantic knowledge base, the similar codebook is determined based on the received first digital information and transmitted to the semantic decoder;

[0009] Using the semantic decoder, feature fusion is performed on the similar codebooks and then decoded to obtain the decoded data corresponding to the training data;

[0010] Based on the training data, decoded data, semantic features, and similar codebooks, obtain the processing loss value;

[0011] Based on the processing loss value, update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder.

[0012] Optionally, both the semantic encoder and the semantic decoder are built on the Swin-Transformer architecture, where Swin-Transformer represents a sliding window transformer.

[0013] Optionally, the step of extracting semantic features from the training data, obtaining similar codebooks corresponding to the semantic features from the quantized codebook stored in the semantic knowledge base, and mapping the index of the similar codebook to first digital information for transmission includes:

[0014] The training data is preprocessed to obtain the first intermediate data; the input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks.

[0015] Extract the semantic features of the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale;

[0016] In the quantized codebook stored in the semantic knowledge base, similar codebooks of extracted semantic features at various scales are obtained, and the indices of the similar codebooks are mapped to first digital information for transmission.

[0017] Optionally, extracting the semantic features of the first intermediate data includes:

[0018] Perform at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0019] Optionally, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes:

[0020] After performing dimensional changes on similar codebooks for semantic features at each scale, feature fusion is performed to obtain second intermediate data; the dimensional changes include at least one of feature channel dimensional changes and size dimensional changes;

[0021] Extract the semantic features of the second intermediate data;

[0022] A convolution operation is performed on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0023] Optionally, extracting the semantic features of the second intermediate data includes:

[0024] Perform at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

[0025] Optionally, dimensionality changes can be performed on similar codebooks for semantic features at each scale, including:

[0026] Perform a change operation to modify the dimensions of similar codebooks for semantic features at each scale; the change operation includes: convolution operation, or convolution operation and interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

[0027] Optionally, based on the processing loss value, the quantization codebook parameters of the semantic knowledge base are updated, including:

[0028] Based on the processing loss value, the quantized codebook parameters of the semantic knowledge base are updated using the exponential moving average (EMA) method.

[0029] Optionally, obtaining the processing loss value based on the training data, decoded data, semantic features, and similar codebooks includes:

[0030] Based on the training data and the decoded data, the reconstruction loss value is obtained;

[0031] Based on the semantic features and similar codebooks, a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder are obtained.

[0032] Based on the reconstruction loss value, the first loss value, and the second loss value, the processing loss value is obtained.

[0033] This application also provides an information processing apparatus, including:

[0034] The first processing module is used to extract semantic features from the training data using a semantic encoder, obtain similar codebooks corresponding to the semantic features from the quantized codebooks stored in the semantic knowledge base, and map the index of the similar codebooks to first digital information for transmission.

[0035] The second processing module is used to determine the similar codebook based on the received first digital information using a semantic knowledge base and transmit it to the semantic decoder.

[0036] The third processing module is used to use the semantic decoder to perform feature fusion on the similar codebook and then decode it to obtain the decoded data corresponding to the training data.

[0037] The first acquisition module is used to acquire the processing loss value based on the training data, decoded data, semantic features, and similar codebooks;

[0038] The first update module is used to update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder based on the processing loss value.

[0039] Optionally, both the semantic encoder and the semantic decoder are built on the Swin-Transformer architecture, where Swin-Transformer represents a sliding window transformer.

[0040] Optionally, the step of extracting semantic features from the training data, obtaining similar codebooks corresponding to the semantic features from the quantized codebook stored in the semantic knowledge base, and mapping the index of the similar codebook to first digital information for transmission includes:

[0041] The training data is preprocessed to obtain the first intermediate data; the input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks.

[0042] Extract the semantic features of the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale;

[0043] In the quantized codebook stored in the semantic knowledge base, similar codebooks of extracted semantic features at various scales are obtained, and the indices of the similar codebooks are mapped to first digital information for transmission.

[0044] Optionally, extracting the semantic features of the first intermediate data includes:

[0045] Perform at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0046] Optionally, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes:

[0047] After performing dimensional changes on similar codebooks for semantic features at each scale, feature fusion is performed to obtain second intermediate data; the dimensional changes include at least one of feature channel dimensional changes and size dimensional changes;

[0048] Extract the semantic features of the second intermediate data;

[0049] A convolution operation is performed on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0050] Optionally, extracting the semantic features of the second intermediate data includes:

[0051] Perform at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

[0052] Optionally, dimensionality changes can be performed on similar codebooks for semantic features at each scale, including:

[0053] Perform a change operation to modify the dimensions of similar codebooks for semantic features at each scale; the change operation includes: convolution operation, or convolution operation and interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

[0054] Optionally, based on the processing loss value, the quantization codebook parameters of the semantic knowledge base are updated, including:

[0055] Based on the processing loss value, the quantized codebook parameters of the semantic knowledge base are updated using the exponential moving average (EMA) method.

[0056] Optionally, obtaining the processing loss value based on the training data, decoded data, semantic features, and similar codebooks includes:

[0057] Based on the training data and the decoded data, the reconstruction loss value is obtained;

[0058] Based on the semantic features and similar codebooks, a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder are obtained.

[0059] Based on the reconstruction loss value, the first loss value, and the second loss value, the processing loss value is obtained.

[0060] This application embodiment also provides an information processing device, including: a processor;

[0061] The processor is configured to use a semantic encoder to extract semantic features from training data, obtain similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and map the index of the similar codebook to first digital information for transmission.

[0062] Using a semantic knowledge base, the similar codebook is determined based on the received first digital information and transmitted to the semantic decoder;

[0063] Using the semantic decoder, feature fusion is performed on the similar codebooks and then decoded to obtain the decoded data corresponding to the training data;

[0064] Based on the training data, decoded data, semantic features, and similar codebooks, obtain the processing loss value;

[0065] Based on the processing loss value, update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder.

[0066] Optionally, both the semantic encoder and the semantic decoder are built on the Swin-Transformer architecture, where Swin-Transformer represents a sliding window transformer.

[0067] Optionally, the step of extracting semantic features from the training data, obtaining similar codebooks corresponding to the semantic features from the quantized codebook stored in the semantic knowledge base, and mapping the index of the similar codebook to first digital information for transmission includes:

[0068] The training data is preprocessed to obtain the first intermediate data; the input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks.

[0069] Extract the semantic features of the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale;

[0070] In the quantized codebook stored in the semantic knowledge base, similar codebooks of extracted semantic features at various scales are obtained, and the indices of the similar codebooks are mapped to first digital information for transmission.

[0071] Optionally, extracting the semantic features of the first intermediate data includes:

[0072] Perform at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0073] Optionally, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes:

[0074] After performing dimensional changes on similar codebooks for semantic features at each scale, feature fusion is performed to obtain second intermediate data; the dimensional changes include at least one of feature channel dimensional changes and size dimensional changes;

[0075] Extract the semantic features of the second intermediate data;

[0076] A convolution operation is performed on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0077] Optionally, extracting the semantic features of the second intermediate data includes:

[0078] Perform at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

[0079] Optionally, dimensionality changes can be performed on similar codebooks for semantic features at each scale, including:

[0080] Perform a change operation to modify the dimensions of similar codebooks for semantic features at each scale; the change operation includes: convolution operation, or convolution operation and interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

[0081] Optionally, based on the processing loss value, the quantization codebook parameters of the semantic knowledge base are updated, including:

[0082] Based on the processing loss value, the quantized codebook parameters of the semantic knowledge base are updated using the exponential moving average (EMA) method.

[0083] Optionally, obtaining the processing loss value based on the training data, decoded data, semantic features, and similar codebooks includes:

[0084] Based on the training data and the decoded data, the reconstruction loss value is obtained;

[0085] Based on the semantic features and similar codebooks, a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder are obtained.

[0086] Based on the reconstruction loss value, the first loss value, and the second loss value, the processing loss value is obtained.

[0087] This application also provides an information processing device, including a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the above-described information processing method.

[0088] This application also provides a readable storage medium storing a program thereon, which, when executed by a processor, implements the steps in the information processing method described above.

[0089] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the information processing method described above.

[0090] The beneficial effects of the above technical solution in this application are as follows:

[0091] In the above scheme, the information processing method utilizes a semantic encoder to extract semantic features from the training data, and obtains similar codebooks corresponding to the semantic features from the quantized codebook stored in the semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission. Using the semantic knowledge base, the similar codebooks are determined based on the received first digital information and transmitted to the semantic decoder. The semantic decoder performs feature fusion on the similar codebooks and then decodes them to obtain the decoded data corresponding to the training data. A processing loss value is obtained based on the training data, decoded data, semantic features, and similar codebooks. Based on the processing loss value, the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder are updated. This method can support the training of a semantic communication system using digital communication methods. This system is compatible with current digital communication, easy to deploy in practice, and effectively solves the problem of difficult practical deployment of existing semantic communication systems. Attached Figure Description

[0092] Figure 1 This is a schematic flowchart of the information processing method according to an embodiment of this application;

[0093] Figure 2 This is a schematic diagram illustrating the implementation framework of the information processing method in an embodiment of this application;

[0094] Figure 3 This is a schematic diagram of the semantic encoder and quantization codebook architecture in an embodiment of this application;

[0095] Figure 4 This is a schematic diagram of the Swin-Transformer block architecture according to an embodiment of this application;

[0096] Figure 5 This is a schematic diagram of the semantic decoder architecture according to an embodiment of this application;

[0097] Figure 6 This is a schematic diagram of the information processing device structure according to an embodiment of this application;

[0098] Figure 7 This is a schematic diagram of the information processing device structure according to an embodiment of this application. Detailed Implementation

[0099] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0100] This application addresses the problem of practical deployment difficulties in existing semantic communication systems by providing an information processing method, such as... Figure 1 As shown, it includes:

[0101] Step 11: Using a semantic encoder, extract semantic features from the training data, obtain similar codebooks corresponding to the semantic features from the quantized codebooks stored in the semantic knowledge base, and map the indexes of the similar codebooks to first digital information for transmission.

[0102] Step 12: Using a semantic knowledge base, determine the similar codebook based on the received first digital information and transmit it to the semantic decoder;

[0103] Step 13: Using the semantic decoder, perform feature fusion on the similar codebook and then decode it to obtain the decoded data corresponding to the training data;

[0104] Step 14: Obtain the processing loss value based on the training data, decoded data, semantic features, and similar codebooks;

[0105] Step 15: Update the model parameters of the semantic encoder, the quantization codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder based on the processing loss value.

[0106] In this scheme, steps 11 to 15 can be executed at least once until the processing loss value indicates that the corresponding loss function converges, but this is not a limitation.

[0107] The information processing method provided in this application embodiment utilizes a semantic encoder to extract semantic features from training data, and obtains similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission. Using the semantic knowledge base, the similar codebooks are determined based on the received first digital information and transmitted to a semantic decoder. The semantic decoder performs feature fusion on the similar codebooks and then decodes them to obtain decoded data corresponding to the training data. A processing loss value is obtained based on the training data, decoded data, semantic features, and similar codebooks. The model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder are updated based on the processing loss value. This method can support the training of a semantic communication system using digital communication methods. This system is compatible with current digital communication, easy to deploy in practice, and effectively solves the problem of difficult practical deployment of semantic communication systems in the prior art.

[0108] Both the semantic encoder and semantic decoder are built on the Swin-Transformer architecture, where Swin-Transformer stands for sliding window transformer. This allows for the simultaneous training of the quantized codebook and efficient semantic feature extraction and transmission.

[0109] In this embodiment, the step of extracting semantic features from training data, obtaining similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission includes: performing input preprocessing on the training data to obtain first intermediate data; the input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks; extracting semantic features from the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale; obtaining similar codebooks (corresponding to) the extracted semantic features at each scale from the quantized codebook stored in the semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission. This specifically realizes the acquisition and transmission of the first digital information. The "image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks" can be performed sequentially (i.e., cascaded processing), but are not limited to this. In this scheme, the step of mapping the index of the similar codebook to the first digital information for transmission may include: when the semantic features of the first intermediate data include semantic features of multiple scales, concatenating the indices of the similar codebooks corresponding to the semantic features of each scale and performing bit conversion to obtain the first digital information, but is not limited thereto.

[0110] The extraction of semantic features from the first intermediate data includes: performing at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale. This allows for the extraction of semantic features from the first intermediate data. While the first extraction operation can be performed multiple times in a cascaded manner, it is not a limitation.

[0111] In this embodiment, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes: performing dimensional changes on the similar codebooks for semantic features at each scale, followed by feature fusion to obtain second intermediate data; the dimensional changes include at least one of feature channel dimension change and size dimension change; extracting the semantic features of the second intermediate data; and performing a convolution operation on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data. This allows for the accurate acquisition of the decoded data corresponding to the training data.

[0112] The extraction of semantic features from the second intermediate data includes: performing at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on a Swin-Transformer block. This allows for the extraction of semantic features from the second intermediate data. While the second extraction operation can be executed multiple times in a cascading manner, it is not a limitation.

[0113] In this embodiment, dimensionality changes are performed on similar codebooks of semantic features at various scales, including: performing a change operation to change the dimensions of similar codebooks of semantic features at various scales; the change operation includes: a convolution operation, or a convolution operation and an interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different. This allows for the specific implementation of dimensionality changes for similar codebooks.

[0114] The step of updating the quantized codebook parameters of the semantic knowledge base based on the processing loss value includes: updating the quantized codebook parameters of the semantic knowledge base using the exponential moving average (EMA) method based on the processing loss value. This can help solve problems such as difficulties in backpropagation during model training and decreased model accuracy due to quantization errors when quantizing features extracted from semantic information.

[0115] In this embodiment, obtaining the processing loss value based on the training data, decoded data, semantic features, and similar codebooks includes: obtaining a reconstruction loss value based on the training data and decoded data; obtaining a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder based on the semantic features and similar codebooks; and obtaining the processing loss value based on the reconstruction loss value, the first loss value, and the second loss value. This allows for the specific acquisition of the processing loss value. The third loss value can represent freezing the quantized codebook and updating the encoder parameters during optimization.

[0116] The information processing method provided in the embodiments of this application will be illustrated with examples below.

[0117] To address the aforementioned technical problems, this application provides an information processing method, specifically a digital semantic communication system design method based on a vector quantization Swin (Shifted Window)-Transformer. Specifically, (1) to accurately extract semantic information and quantization representation, a backbone neural network combining a Swin-Transformer and a Vector Quantized Variational Autoencoder (VQ-VAE) architecture is designed to effectively extract semantic features while training the quantized codebook, achieving accurate semantic feature extraction and efficient transmission. (2) To address the error introduced by the limited quantized codebook space, an Exponential Moving Average (EMA) method is introduced to enhance the representational ability of the quantized codebook in order to construct a vector quantized codebook with strong representation capabilities. (3) To assist in the accurate reconstruction of semantic information using the vector quantized codebook, a feature fusion neural network is designed to perform multi-scale fusion of codebook feature vectors at different scales, which is then input into a decoder constructed based on Swin-Transformer blocks to achieve efficient semantic decoding and reconstruction.

[0118] Specifically, the structural framework of the digital semantic communication system based on vector quantization Swin-Transformer designed in this scheme can be as follows: Figure 2As shown, the main modules include: a semantic encoder, a quantization codebook (i.e., a semantic knowledge base), and a semantic decoder. The transmitting and receiving link modules can be built based on current digital communication system processing modules. The semantic encoder can use a Swin-Transformer and VQ-VAE architecture as the backbone network to extract semantic features from the information source (corresponding to the training data mentioned above), and simultaneously train the quantization codebook (which can correspond to using the semantic encoder to extract semantic features from the training data, obtaining similar codebooks corresponding to the semantic features from the quantization codebook stored in the semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission). The transmitting link (module) can perform baseband signal processing on the semantic encoder output vector, which may include symbol modulation (e.g., Quadrature Amplitude Modulation, QAM), OFDM (Orthogonal Frequency Division Multiplexing) shaping, etc. After wireless channel fading, the receiving link (module) can perform inverse transformation of the received signal, which may include OFDM reception, channel estimation and equalization, demodulation, etc. The information obtained from the inverse transform (which may correspond to the index of the aforementioned similar codebook) is input into the quantized codebook to obtain the corresponding feature vector (which may correspond to the aforementioned use of a semantic knowledge base to determine the similar codebook based on the received first digital information and transmit it to the semantic decoder). After feature fusion, the source is decoded and reconstructed through the semantic decoder to obtain the destination (which may correspond to the aforementioned use of the semantic decoder to perform feature fusion on the similar codebook and then decode it to obtain the decoded data corresponding to the training data). The semantic decoder can also be built based on the Swin-Transformer architecture; this may correspond to the aforementioned semantic encoder and semantic decoder both being built based on the Swin-Transformer architecture, where Swin-Transformer represents a sliding window transformer.

[0119] Among them, targeting Figure 2 The structural framework shown may include the following specific data processing flow:

[0120] (1) Assuming the input source s is encoded by a semantic encoder, the following is obtained: (This can correspond to the first numerical information mentioned above), that is:

[0121]

[0122] In the formula, E θ (·) represents the semantic encoder model, and θ represents the model parameters.

[0123] (2) Information is obtained through processing by the sending end link module. In the formula, T x (·) indicates the transmitting end link processing. Considering the wireless fading channel, we get:

[0124] y=h⊙x+n (2)

[0125] In the formula, ⊙ represents the dot product, h represents the channel impulse response, and n represents independent and identically distributed noise that follows a Gaussian distribution, i.e. in, Indicates noise power.

[0126] (3) Information is obtained after processing by the receiving end link. Input quantization codebook C χ (·) Mapping yields the feature vector (This can correspond to the similar codebooks mentioned above), that is Among them, R x This indicates the receiver link processing, and χ represents the quantization codebook parameters. The input semantic decoder realizes the decoding and reconstruction of semantic information, that is:

[0127]

[0128] In the formula, This represents a semantic decoder model. Indicates model parameters, This corresponds to the decoded data mentioned above.

[0129] Based on the above, it can be seen that this solution can optimize the model parameters of the designed semantic communication system. Optimization (which can correspond to updating the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder as described above) minimizes the reconstruction error of the information source, i.e.:

[0130]

[0131] In the formula, p s This represents the true distribution of the input data s. Indicates s and Distortion between them can be defined as Indicates the optimized

[0132] The following sections will introduce each part of this plan.

[0133] I. Semantic Encoder and Quantization Codebook Construction;

[0134] The semantic encoder and quantization codebook architecture design based on vector quantization Swin-Transformer is as follows: Figure 3 As shown, the image is used as the input source, i.e. Where H represents the image height, W represents the image width, and C represents the number of (feature) channels (C1, C2, C3, and C4 represent the number of channels at different stages). The example in the figure shows the number of channels for the input source as 3. Represents real numbers. Specifically, based on... Figure 3 The architecture shown can process information sources in the following ways:

[0135] 1) First, image input preprocessing is achieved through a cascaded approach using three modules: Patch Partition, Linear Embedding, and Swin Transformer Block. Specifically, Patch Partition and Linear Embedding map the image into tokens (minimum processing units / minimum image blocks) for a self-attention mechanism, transforming [H,W,3] into [H / 2,W / 2,C1]. The Swin Transformer Block performs preliminary feature fusion. This corresponds to the input preprocessing of the training data described above, yielding the first intermediate data. The input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on the Swin Transformer Block.

[0136] 2) Then, semantic features are extracted step-by-step through a combination of three structurally similar modules: the Patch Merging module and the Swin-Transformer Block module shown in the diagram. N1, N2, and N3 represent the cumulative number of cascaded Patch Merging and Swin-Transformer Block modules, for example, N1 = 1. The Patch Merging module can be implemented using two-dimensional convolution operations, achieving downsampling by setting the kernel size and stride. Therefore, after passing through the Patch Merging layer, the height and width of the feature map are halved, while the number of channels doubles. Specifically, the structural diagram of the Swin-Transformer Block is shown below. Figure 4As shown, the main approach employs Windows Multi-Head Self-Attention (W-MSA) and Shifted Windows Multi-Head Self-Attention (SW-MSA) to achieve efficient semantic feature extraction while significantly reducing computational complexity. The W-MSA structure limits the scope of self-attention computation, changing it from global computation to computation only within a given window size, effectively reducing computational load. Simultaneously, SW-MSA enables information transfer between different windows. This part corresponds to the extraction of semantic features from the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale; the extraction of semantic features from the first intermediate data includes: performing at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on Windows Multi-Head Self-Attention (W-MSA) and Shifted Windows Multi-Head Self-Attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0137] in, Figure 3 The feature signal processing flow in the architecture shown can be as follows:

[0138]

[0139] In the formula, W-MSA(·) represents window multi-head attention operation, SW-MSA(·) represents sliding window multi-head attention operation, MLP(·) represents fully connected operation, LN(·) represents layer normalization operation, and z l-1 This represents the input to the Swin-Transformer block. z l , z represents an intermediate quantity during the calculation process. l+1 This represents the output of the Swing-Transformer block. This scheme allows for the extraction of semantic features across different dimensions by controlling the sliding window size and the number of cascaded Swing-Transformer blocks.

[0140] In addition, based on Figure 3 The architecture shown extracts semantic features (i.e., feature vectors) from the three combinations (Patch Merging module and Swin-TransformerBlock module). and The mapping is to a quantized codebook, that is... and Where N represents the number of quantization vectors (i.e., the number of rows in the quantization codebook), K represents the dimension of each quantization vector (i.e., the number of columns in the quantization codebook), E represents the quantization codebook, and E1 represents... Corresponding similar codebook (i.e., the quantized codebook obtained through mapping), E2 representation Corresponding similar codebooks, E3 representation The corresponding similar codebooks; these can correspond to the similar codebooks of semantic features extracted at various scales obtained from the quantized codebooks stored in the semantic knowledge base mentioned above. It should be noted that... Figure 3 The diagram shown is a three-tier architecture, for example only. In actual implementation, it can be flexibly expanded as needed. This solution... and In the vector quantization codebook mapping, mapping indices are obtained respectively (an example of the index of the similar codebooks mentioned above), i.e. and m1 represents The total number of corresponding similar codebooks, m2 represents The total number of sequence numbers of the corresponding similar codebooks, m3 represents The total number of corresponding similar codebook indices, where I1 represents the index of E1, I2 represents the index of E2, and I3 represents the index of E3. The feature extraction vector and the quantized codebook can be floating-point data, mapped to the index I using bit transmission.

[0141] The mapping from semantic features to the quantization vectors of the quantization codebook can be determined by calculating the Euclidean distance, for example, using the following formula:

[0142]

[0143] In formula (9) This represents the quantization vector of each row of the quantization codebook obtained from the semantic features (the quantization vector e of the j-th row). j Given a Euclidean distance (j ≥ 1 and ≤ N), output the quantized vector e of the row number k with the smallest distance. 1,k ,k, from which the quantization codebook E1 is obtained; the value of k is not fixed (it can be one or more); formulas (10) and (11) are similar to formula (9), and will not be repeated here. It should be noted that the quantization codebook can contain multiple quantization vectors, and each row in the quantization codebook can be used as a quantization vector for similarity comparison, but this is not a limitation.

[0144] 3) Finally, the mapping numbers I1, I2 and I3 are concatenated to form a matrix, and then mapped to a bit sequence for transmission; that is, the Concatenate & Bit Conversion in the diagram is executed; which corresponds to the above-mentioned mapping of the index of the similar codebook to the first digital information for transmission.

[0145] II. Quantization codebook update;

[0146] Figure 3 In the architecture shown, the quantized codebook parameters need to be updated simultaneously with the semantic encoder and semantic decoder. The current codebook update has the following problems: (1) Quantization non-differentiability. Specifically, the input feature vector is mapped to the nearest embedding vector in the codebook (through the argmin operation). Since argmin is non-differentiable, the gradient cannot be directly backpropagated to the codebook vector through the quantization step, causing the traditional gradient descent method to fail. (2) Insufficient codebook representation capability, that is, some embedding vectors may be rarely used due to poor initialization or insufficient training, resulting in wasted codebook capacity.

[0147] To address the above issues, this solution introduces the Exponential Moving Average (EMA) method based on the Straight-through Estimator (ST) method to improve the overall utilization and representational capability of the codebook. Corresponding to the aforementioned processing loss value, the EMA method is used to update the quantized codebook parameters of the semantic knowledge base. The core idea of ​​EMA is to achieve an online approximation of the global mean by weighted averaging of historical statistics and current training batch information, specifically implemented using the following formula:

[0148] For the epoch-th iteration, the extracted semantic features are mapped to the quantization vector {e} of the quantization codebook. i The cumulative number of the i = 1, 2, ..., N is:

[0149]

[0150] In the formula, This indicates mapping to the quantization vector e. i The number of semantic feature vectors, where γ∈[0,1) represents the decay rate.

[0151] Wherein, mapped to the quantization vector {e i The cumulative vector sum of semantic features for the group i = 1, 2, ..., N is:

[0152]

[0153] In the formula, This represents the cumulative vector sum mentioned above.

[0154] Based on the above, the quantization vector e i It can be updated using the following formula:

[0155]

[0156] In the formula, parameter ∈ = 10 -5.

[0157] III. Construction of Feature Fusion Neural Network and Semantic Decoder;

[0158] In this scheme, the semantic decoder architecture design including a Feature Fusion Network (FFN) can be as follows: Figure 5 As shown, for features of different scales obtained from the above bit sequence The feature channel dimension and size dimension can be changed by using convolution modules {ConV1, ConV2, ConV3} and interpolation modules {Interpolate1, Interpolate2} respectively, and then feature fusion can be achieved by summing. This can be performed on similar codebooks of semantic features at each scale after performing dimension changes, resulting in second intermediate data. The dimension changes include at least one of feature channel dimension changes and size dimension changes. Specifically, performing dimension changes on similar codebooks of semantic features at each scale includes performing a change operation to change the dimensions of similar codebooks of semantic features at each scale. The change operation includes convolution operations, or convolution operations and interpolation operations. The change operations for semantic features at different scales may be the same or different. and correspond, and correspond, and correspond.

[0159] Subsequently, this can be achieved through a combination of three structurally similar components: an Up-sampling module and a Swin-Transformer Block (which can be similar in specific terms). Figure 2 The semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA) is used to gradually extract semantic features. This corresponds to the extraction of semantic features from the second intermediate data. The extraction of semantic features from the second intermediate data includes: performing at least one second extraction operation to obtain the semantic features of the second intermediate data. The second extraction operation includes: upsampling operation and semantic feature extraction operation based on Swin-Transformer block. N4, N5, and N6 represent the number of cumulative cascaded Up-sampling modules and Swin-Transformer Block modules, for example, N4 = 1 and N5 = 2. In this scheme, the Up-sampling module can be implemented using two-dimensional deconvolution operation, and upsampling is achieved by setting the kernel size and stride. Therefore, after the Up-sampling layer, the channels and size of the feature map will double; the parameters in the figure can be found in [reference needed]. Figure 3The meanings of relevant parameters, such as C5, C6, and C7 representing the number of channels at different stages, will not be elaborated here. Among them, "gradually extracting semantic features" may include: after passing through the feature fusion network, passing through 3 upsampling layers and the Swin-Transformer Block to gradually increase the image feature size and reduce the feature channel dimension, but it is not limited to this.

[0160] Finally, the size of the signal source can be matched by the ConV4 convolution operation; the convolution operation can be performed on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0161] IV. Offline Training and Online Deployment;

[0162] This approach uses DIV2K (DIV2K Benchmark for High-Resolution Image and Video Processing) as the training dataset. This dataset contains high-resolution images and videos, typically with resolutions as high as 2K (2048×1080 pixels), 4K (3840×2160 pixels), or even 8K (7680×4320 pixels). During training, images are randomly cropped to 256×256 pixels. This approach targets... Figures 3 to 5 The loss function used for model training in the example architecture can be as follows:

[0163]

[0164] In the formula, The first term represents the total loss, and sg[·] represents the gradient stopping operation; the second term represents the reconstruction loss; the third term is used to optimize the quantized codebook; the third term represents freezing the quantized codebook and updating the encoder parameters during optimization; where, The above indicates The above indicates The above indicates The second and third items can correspond to VQ-VAE. This part can correspond to the above-mentioned method of obtaining processing loss values ​​based on the training data, decoding data, semantic features, and similar codebooks; updating the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder based on the processing loss values; wherein, obtaining processing loss values ​​based on the training data, decoding data, semantic features, and similar codebooks includes: obtaining reconstruction loss values ​​based on the training data and decoding data (corresponding to the first item in the above formula (15)); obtaining a first loss value for updating the quantized codebook based on the semantic features and similar codebooks (corresponding to the second item in the above formula (15)) and a second loss value for updating the model parameters of the semantic encoder based on the semantic features and similar codebooks (corresponding to the third item in the above formula (15)); obtaining processing loss values ​​based on the reconstruction loss value, the first loss value, and the second loss value (corresponding to the third item in the above formula (15)). ).

[0165] Based on the above, the digital semantic communication system design scheme based on vector quantization Swin-Transformer proposed in this application may include an offline training phase and an online deployment phase, as illustrated in the following example:

[0166] 1. Offline training phase:

[0167] Input: Training dataset DIV2K, randomly cropped to 256×256.

[0168] Initialization: Randomly initialize the semantic encoder parameters θ, quantization codebook parameters χ, and semantic decoder parameters.

[0169] Training: Update model parameters using backpropagation based on the loss function of equation (15) Until the loss function converges; corresponding to the above, based on the training data, decoding data, semantic features, and similar codebooks, the processing loss value is obtained; based on the processing loss value, the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder are updated.

[0170] Output: θ, χ

[0171] 2. Online deployment phase:

[0172] Deployment: At the sending end, deploy the semantic encoder and its trained parameters θ, and the quantization codebook and its trained parameters χ. Similarly, at the receiving end, deploy the semantic decoder and its trained parameters. The quantization codebook and its trained parameters χ.

[0173] Input: Images of any resolution, similar to the training data described above.

[0174] Output: The reconstructed image, which may be similar to the decoded data described above.

[0175] As described above, the digital semantic communication system design scheme based on vector quantization Swin-Transformer provided in this application, by designing a backbone neural network that combines Swin-Transformer and vector quantization variational autoencoder architecture, can effectively extract semantic features while training the quantization codebook, thereby achieving efficient semantic feature extraction and transmission. By introducing the quantization codebook, a high compression ratio and reconstruction performance of semantic features can be achieved. At the same time, this scheme is compatible with current digital communication and is easy to deploy in practice.

[0176] In summary, the embodiments of this application provide a digital semantic communication system that is compatible with current digital communication links, based on the semantic communication paradigm, an important method for the future integration of AI (artificial intelligence) and communication.

[0177] This application also provides an information processing device, such as... Figure 6 As shown, it includes:

[0178] The first processing module 61 is used to extract semantic features from the training data using a semantic encoder, obtain similar codebooks corresponding to the semantic features from the quantized codebooks stored in the semantic knowledge base, and map the index of the similar codebooks to first digital information for transmission.

[0179] The second processing module 62 is used to determine the similar codebook based on the received first digital information using a semantic knowledge base and transmit it to the semantic decoder.

[0180] The third processing module 63 is used to use the semantic decoder to perform feature fusion on the similar codebook and then decode it to obtain the decoded data corresponding to the training data.

[0181] The first acquisition module 64 is used to acquire the processing loss value based on the training data, decoded data, semantic features and similar codebooks;

[0182] The first update module 65 is used to update the model parameters of the semantic encoder, the quantization codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder according to the processing loss value.

[0183] The information processing apparatus provided in this application embodiment utilizes a semantic encoder to extract semantic features from training data, and obtains similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission. Using the semantic knowledge base, the similar codebooks are determined based on the received first digital information and transmitted to a semantic decoder. The semantic decoder performs feature fusion on the similar codebooks and then decodes them to obtain decoded data corresponding to the training data. A processing loss value is obtained based on the training data, decoded data, semantic features, and similar codebooks. The model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder are updated based on the processing loss value. This apparatus can support the training of a semantic communication system using digital communication methods. This system is compatible with current digital communication, easy to deploy in practice, and effectively solves the problem of difficult practical deployment of existing semantic communication systems.

[0184] The semantic encoder and semantic decoder are both built on the Swin-Transformer architecture, where Swin-Transformer stands for sliding window transformer.

[0185] In this embodiment, the step of extracting semantic features from training data, obtaining similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission includes: performing input preprocessing on the training data to obtain first intermediate data; the input preprocessing includes: image block segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks; extracting semantic features from the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale; obtaining similar codebooks of the extracted semantic features at each scale from the quantized codebook stored in the semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission.

[0186] The extraction of semantic features from the first intermediate data includes: performing at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0187] In this embodiment of the application, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes: performing dimensional changes on the similar codebooks of semantic features at each scale and then performing feature fusion to obtain second intermediate data; the dimensional changes include at least one of feature channel dimension changes and size dimension changes; extracting the semantic features of the second intermediate data; and performing a convolution operation on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0188] The step of extracting the semantic features of the second intermediate data includes: performing at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

[0189] In this embodiment of the application, dimensional changes are performed on similar codebooks of semantic features at each scale, including: performing a change operation to change the dimensions of similar codebooks of semantic features at each scale; the change operation includes: a convolution operation, or a convolution operation and an interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

[0190] The step of updating the quantized codebook parameters of the semantic knowledge base based on the processing loss value includes: updating the quantized codebook parameters of the semantic knowledge base using the exponential moving average (EMA) method based on the processing loss value.

[0191] In this embodiment of the application, obtaining the processing loss value based on the training data, decoding data, semantic features, and similar codebooks includes: obtaining a reconstruction loss value based on the training data and decoding data; obtaining a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder based on the semantic features and similar codebooks; and obtaining the processing loss value based on the reconstruction loss value, the first loss value, and the second loss value.

[0192] The implementation embodiments of the above information processing method are all applicable to the embodiments of the information processing device and can achieve the same technical effect.

[0193] This application also provides an information processing device, such as... Figure 7 As shown, it includes: processor 71;

[0194] The processor 71 is used to extract semantic features from training data using a semantic encoder, obtain similar codebooks corresponding to the semantic features from the quantized codebooks stored in the semantic knowledge base, and map the index of the similar codebooks to first digital information for transmission.

[0195] Using a semantic knowledge base, the similar codebook is determined based on the received first digital information and transmitted to the semantic decoder;

[0196] Using the semantic decoder, feature fusion is performed on the similar codebooks and then decoded to obtain the decoded data corresponding to the training data;

[0197] Based on the training data, decoded data, semantic features, and similar codebooks, obtain the processing loss value;

[0198] Based on the processing loss value, update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder.

[0199] The information processing device provided in this application embodiment utilizes a semantic encoder to extract semantic features from training data, and obtains similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission. Using the semantic knowledge base, the similar codebooks are determined based on the received first digital information and transmitted to a semantic decoder. The semantic decoder performs feature fusion on the similar codebooks and then decodes them to obtain decoded data corresponding to the training data. A processing loss value is obtained based on the training data, decoded data, semantic features, and similar codebooks. The model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder are updated based on the processing loss value. This device can support the training of a semantic communication system using digital communication methods. This system is compatible with current digital communication, easy to deploy in practice, and effectively solves the problem of difficult practical deployment of semantic communication systems in the prior art.

[0200] The semantic encoder and semantic decoder are both built on the Swin-Transformer architecture, where Swin-Transformer stands for sliding window transformer.

[0201] In this embodiment, the step of extracting semantic features from training data, obtaining similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission includes: performing input preprocessing on the training data to obtain first intermediate data; the input preprocessing includes: image block segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks; extracting semantic features from the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale; obtaining similar codebooks of the extracted semantic features at each scale from the quantized codebook stored in the semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission.

[0202] The extraction of semantic features from the first intermediate data includes: performing at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

[0203] In this embodiment of the application, the step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes: performing dimensional changes on the similar codebooks of semantic features at each scale and then performing feature fusion to obtain second intermediate data; the dimensional changes include at least one of feature channel dimension changes and size dimension changes; extracting the semantic features of the second intermediate data; and performing a convolution operation on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

[0204] The step of extracting the semantic features of the second intermediate data includes: performing at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

[0205] In this embodiment of the application, dimensional changes are performed on similar codebooks of semantic features at each scale, including: performing a change operation to change the dimensions of similar codebooks of semantic features at each scale; the change operation includes: a convolution operation, or a convolution operation and an interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

[0206] The step of updating the quantized codebook parameters of the semantic knowledge base based on the processing loss value includes: updating the quantized codebook parameters of the semantic knowledge base using the exponential moving average (EMA) method based on the processing loss value.

[0207] In this embodiment of the application, obtaining the processing loss value based on the training data, decoding data, semantic features, and similar codebooks includes: obtaining a reconstruction loss value based on the training data and decoding data; obtaining a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder based on the semantic features and similar codebooks; and obtaining the processing loss value based on the reconstruction loss value, the first loss value, and the second loss value.

[0208] The implementation embodiments of the above information processing method are all applicable to the embodiments of the information processing device and can achieve the same technical effect.

[0209] This application also provides an information processing device, including a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the above-described information processing method.

[0210] The implementation embodiments of the above information processing method are all applicable to the embodiments of the information processing device and can achieve the same technical effect.

[0211] This application also provides a readable storage medium storing a program thereon, which, when executed by a processor, implements the steps in the information processing method described above.

[0212] The implementation embodiments of the above information processing method are all applicable to the embodiments of the readable storage medium and can achieve the same technical effect.

[0213] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described information processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0214] It should be noted that many of the functional components described in this specification are referred to as modules in order to more specifically emphasize the independence of their implementation.

[0215] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0216] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0217] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0218] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An information processing method, characterized in that, include: Using a semantic encoder, semantic features are extracted from the training data, and similar codebooks corresponding to the semantic features are obtained from the quantized codebooks stored in the semantic knowledge base. The indexes of the similar codebooks are mapped to first digital information for transmission. Using a semantic knowledge base, the similar codebook is determined based on the received first digital information and transmitted to the semantic decoder; Using the semantic decoder, feature fusion is performed on the similar codebooks and then decoded to obtain the decoded data corresponding to the training data; Based on the training data, decoded data, semantic features, and similar codebooks, obtain the processing loss value; Based on the processing loss value, update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder.

2. The information processing method according to claim 1, characterized in that, Both the semantic encoder and semantic decoder are built on the Swin-Transformer architecture, where Swin-Transformer stands for sliding window transformer.

3. The information processing method according to claim 1 or 2, characterized in that, The step of extracting semantic features from training data, obtaining similar codebooks corresponding to the semantic features from the quantized codebook stored in the semantic knowledge base, and mapping the indices of the similar codebooks to first digital information for transmission includes: The training data is preprocessed to obtain the first intermediate data; the input preprocessing includes: image patch segmentation, linear embedding, and feature fusion based on Swin-Transformer blocks. Extract semantic features from the first intermediate data; the semantic features of the first intermediate data include semantic features at at least one scale; In the quantized codebook stored in the semantic knowledge base, similar codebooks of extracted semantic features at various scales are obtained, and the indices of the similar codebooks are mapped to first digital information for transmission.

4. The information processing method according to claim 3, characterized in that, The extraction of semantic features from the first intermediate data includes: Perform at least one first extraction operation to obtain the semantic features of the first intermediate data; the first extraction operation includes: a downsampling operation, and a semantic feature extraction operation based on window multi-head attention (W-MSA) and sliding window multi-head attention (SW-MSA); one first extraction operation can extract semantic features at one scale.

5. The information processing method according to claim 1 or 2, characterized in that, The step of performing feature fusion on the similar codebooks and then decoding to obtain the decoded data corresponding to the training data includes: After performing dimensional changes on similar codebooks for semantic features at each scale, feature fusion is performed to obtain second intermediate data; the dimensional changes include at least one of feature channel dimensional changes and size dimensional changes; Extract the semantic features of the second intermediate data; A convolution operation is performed on the semantic features of the second intermediate data to obtain the decoded data corresponding to the training data.

6. The information processing method according to claim 5, characterized in that, The extraction of semantic features from the second intermediate data includes: Perform at least one second extraction operation to obtain the semantic features of the second intermediate data; the second extraction operation includes: an upsampling operation and a semantic feature extraction operation based on the Swin-Transformer block.

7. The information processing method according to claim 5, characterized in that, Dimensional changes are performed on similar codebooks for semantic features at each scale, including: Perform a change operation to modify the dimensions of similar codebooks for semantic features at each scale; the change operation includes: convolution operation, or convolution operation and interpolation operation; the change operations corresponding to semantic features at different scales may be the same or different.

8. The information processing method according to claim 1, characterized in that, Based on the processing loss value, update the quantized codebook parameters of the semantic knowledge base, including: Based on the processing loss value, the quantized codebook parameters of the semantic knowledge base are updated using the exponential moving average (EMA) method.

9. The information processing method according to claim 1, characterized in that, The step of obtaining the processing loss value based on the training data, decoded data, semantic features, and similar codebooks includes: Based on the training data and the decoded data, the reconstruction loss value is obtained; Based on the semantic features and similar codebooks, a first loss value for updating the quantized codebook and a second loss value for updating the model parameters of the semantic encoder are obtained. Based on the reconstruction loss value, the first loss value, and the second loss value, the processing loss value is obtained.

10. An information processing device, characterized in that, include: The first processing module is used to extract semantic features from the training data using a semantic encoder, obtain similar codebooks corresponding to the semantic features from the quantized codebooks stored in the semantic knowledge base, and map the index of the similar codebooks to first digital information for transmission. The second processing module is used to determine the similar codebook based on the received first digital information using a semantic knowledge base and transmit it to the semantic decoder. The third processing module is used to use the semantic decoder to perform feature fusion on the similar codebook and then decode it to obtain the decoded data corresponding to the training data. The first acquisition module is used to acquire the processing loss value based on the training data, decoded data, semantic features, and similar codebooks; The first update module is used to update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder based on the processing loss value.

11. An information processing device, characterized in that, include: processor; The processor is configured to use a semantic encoder to extract semantic features from training data, obtain similar codebooks corresponding to the semantic features from a quantized codebook stored in a semantic knowledge base, and map the index of the similar codebook to first digital information for transmission. Using a semantic knowledge base, the similar codebook is determined based on the received first digital information and transmitted to the semantic decoder; Using the semantic decoder, feature fusion is performed on the similar codebooks and then decoded to obtain the decoded data corresponding to the training data; Based on the training data, decoded data, semantic features, and similar codebooks, obtain the processing loss value; Based on the processing loss value, update the model parameters of the semantic encoder, the quantized codebook parameters of the semantic knowledge base, and the model parameters of the semantic decoder.

12. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the information processing method as described in any one of claims 1 to 9.

13. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the information processing method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the information processing method as described in any one of claims 1 to 9.

Images