Semantic information transmission method and device

Abstract

The invention provides a semantic information transmission method and device, and the method comprises the steps: receiving a to-be-detected image, carrying out the semantic feature extraction of the to-be-detected image, and obtaining a target semantic feature; performing coding processing on the target semantic feature to obtain a target semantic vector corresponding to the to-be-detected image; inputting the target semantic vector into a semantic importance discrimination model obtained by pre-training, and outputting a target semantic importance value through the processing of the semantic importance discrimination model; inputting the target semantic importance value into a quantization series determination model obtained through pre-training, and outputting a target quantization series through processing of the quantization series determination model; performing quantization coding processing on the target semantic vector according to the target quantization series to obtain a target bit stream; and performing modulation processing on the target bit stream to obtain a target semantic signal, sending the target semantic signal and the target quantization series to a channel, and sending the target semantic signal and the target quantization series to a receiving end through the channel.

Description

Technical Field

[0001] The present disclosure relates to the field of communications, and in particular, to a semantic information transmission method and apparatus. Background Art

[0002] Semantic communication aims at conveying meaning. By extracting the semantic information of the information source and encoding the extracted semantic information and then transmitting it to the receiving end, efficient information transmission is achieved.

[0003] When the current semantic information is transmitted, the transmitted bit stream is classified by the high and low bits of the bits, without considering the semantic importance. Errors in the low-order bits may also have a greater impact on the task performance than errors in the high-order bits. Summary of the Invention

[0004] In view of this, the purpose of the present disclosure is to propose a semantic information transmission method and apparatus to solve or partially solve the above problems.

[0005] Based on the above purpose, the first aspect of the present disclosure provides a semantic information transmission method, which is applied to a sending end, and the method includes:

[0006] Receiving a to-be-detected image, extracting semantic features from the to-be-detected image to obtain target semantic features;

[0007] Performing encoding processing on the target semantic features to obtain a target semantic vector corresponding to the to-be-detected image;

[0008] Inputting the target semantic vector into a pre-trained semantic importance discrimination model, and after being processed by the semantic importance discrimination model, outputting a target semantic importance value;

[0009] Inputting the target semantic importance value into a pre-trained quantization level determination model, and after being processed by the quantization level determination model, outputting a target quantization level;

[0010] Performing quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bit stream;

[0011] Performing modulation processing on the target bit stream to obtain a target semantic signal, and sending the target semantic signal and the target quantization level to a channel, and sending them to a receiving end via the channel.

[0012] Based on the above purpose, the second aspect of the present disclosure provides a semantic information transmission method, which is applied to a receiving end, and the method includes:

[0013] Receiving a target semantic signal and a target quantization level in a channel, and performing demodulation processing on the target semantic signal to obtain a target bit stream;

[0014] Perform inverse quantization processing on the target bitstream using the target quantization level to obtain a target semantic vector;

[0015] Perform decoding processing on the target semantic vector to obtain target semantic features;

[0016] Use the target semantic features to perform object detection on the image to be detected, and obtain the detection position and detection category of the target object.

[0017] Based on the same inventive concept, a third aspect of the present disclosure proposes a semantic information transmission device, which is arranged at the sending end and includes:

[0018] An image receiving module, configured to receive the image to be detected, extract semantic features from the image to be detected, and obtain target semantic features;

[0019] An encoding processing module, configured to perform encoding processing on the target semantic features to obtain a target semantic vector corresponding to the image to be detected;

[0020] A model processing module, configured to input the target semantic vector into a pre-trained semantic importance discrimination model, and output a target semantic importance value after being processed by the semantic importance discrimination model;

[0021] A quantization level determination module, configured to input the target semantic importance value into a pre-trained quantization level determination model, and output a target quantization level after being processed by the quantization level determination model;

[0022] A bitstream determination module, configured to perform quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bitstream;

[0023] A data sending module, configured to perform modulation processing on the target bitstream to obtain a target semantic signal, and send the target semantic signal and the target quantization level to a channel, and send them to the receiving end via the channel.

[0024] Based on the same inventive concept, a fourth aspect of the present disclosure proposes a semantic information transmission device, which is arranged at the receiving end and includes:

[0025] A data receiving module, configured to receive the target semantic signal and the target quantization level in the channel, and perform demodulation processing on the target semantic signal to obtain a target bitstream;

[0026] An inverse quantization processing module, configured to perform inverse quantization processing on the target bitstream using the target quantization level to obtain a target semantic vector;

[0027] A decoding processing module, configured to perform decoding processing on the target semantic vector to obtain a target semantic feature;

[0028] A detection module, configured to perform object detection on the image to be detected by using the target semantic feature, and obtain a detection position and a detection category of a target object.

[0029] Based on the same inventive concept, a fifth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable by the processor. When the processor executes the computer program, the semantic information transmission method described above is implemented.

[0030] Based on the same inventive concept, a sixth aspect of the present disclosure provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the semantic information transmission method described above.

[0031] As can be seen from the above, the present disclosure provides a semantic information transmission method and apparatus. An image to be detected is received, semantic feature extraction is performed on the image to be detected to obtain a target semantic feature. Encoding processing is performed on the target semantic feature to obtain a target semantic vector corresponding to the image to be detected. The target semantic vector is input into a pre-trained semantic importance discrimination model, and after being processed by the semantic importance discrimination model, a target semantic importance value is output. The target semantic importance value is input into a pre-trained quantization level determination model, and after being processed by the quantization level determination model, a target quantization level is output. The corresponding target quantization level is determined through the semantic importance value, that is, subsequent non-uniform quantization of semantic information can be performed according to the semantic importance, improving the transmission efficiency of semantic information. Quantization encoding processing is performed on the target semantic vector according to the target quantization level to obtain a target bit stream, modulation processing is performed on the target bit stream to obtain a target semantic signal, and the target semantic signal and the target quantization level are sent to a channel and transmitted to a receiving end through the channel. By determining the corresponding quantization level according to the semantic importance value and performing quantization processing, it is avoided that when dividing according to high and low bit positions, the data in the low bit positions contains important semantic information, and it is also avoided that important semantic information is lost due to rule-based hard threshold truncation. Description of the Drawings

[0032] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 Flow chart of the semantic information transmission method according to an embodiment of the present disclosure;

[0034] Figure 2 Schematic diagram of the process of separable convolution according to an embodiment of the present disclosure;

[0035] Figure 3 Schematic diagram of the basic component modules of the semantic importance discrimination model in this embodiment;

[0036] Figure 4 Schematic diagram of semantic importance guidance according to an embodiment of the present disclosure;

[0037] Figure 5 Schematic diagram of the selection of quantization levels based on convolution according to an embodiment of the present disclosure;

[0038] Figure 6 Flow chart of the reparameterization according to an embodiment of the present disclosure;

[0039] Figure 7 Flow chart of the semantic information transmission method according to another embodiment of the present disclosure;

[0040] Figure 8 Schematic diagram of the framework of the semantic information transmission method according to another embodiment of the present disclosure;

[0041] Figure 9 Schematic diagram of the straight-through gradient method according to another embodiment of the present disclosure;

[0042] Figure 10 Block diagram of the structure of the semantic information transmission device according to an embodiment of the present disclosure;

[0043] Figure 11 Block diagram of the structure of the semantic information transmission device according to another embodiment of the present disclosure;

[0044] Figure 12 Schematic diagram of the structure of the electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0045] To make the objectives, technical solutions and advantages of the present disclosure more clear and understandable, the present disclosure will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0046] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the field to which the present disclosure belongs. The "first", "second" and similar terms used in the embodiments of the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to represent relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0047] The following are the explanations of the terms related to the present disclosure:

[0048] MobileNetV2: MobileNet V2 is a lightweight image classification and object detection network that can find a balance between high accuracy and low latency.

[0049] SSD: Single Shot MultiBox Detector (SSD). The SSD algorithm is a single-stage method improved on the basis of YOLO. By fusing the bounding boxes (BBs) on multiple feature maps, it improves the detection accuracy while increasing the speed, and its performance exceeds that of YOLO and Faster-rcnn (under the condition of large objects).

[0050] Semantic communication aims at conveying meaning. By extracting the semantic information of the information source and encoding the extracted semantic information and then transmitting it to the receiving end, efficient information transmission is achieved. The current semantic communication framework mainly includes semantic codecs and channel codecs. Semantic codecs are based on neural network structures such as CNN and Transformer, and can extract the deep semantic information of the information source. Channel codecs are based on the Deep Joint Source-Channel Coding (Deep JSCC) technology, combined with technologies such as SNR adaptation and code rate control, and can improve the ability of the communication system to cope with different channel conditions and ensure the semantic task performance at the receiving end.

[0051] In current digital wireless transmission methods for semantic communication, the transmitted bit stream is classified by the high and low bits of the bits to ensure that the high-order bits have a lower error rate. However, such a division does not take into account semantic importance, and errors in the low-order bits may also have a higher impact on task performance than errors in the high-order bits. And by the method of mixing high-order and low-order modulation to transmit the bit stream, an additional selector needs to be designed to select the modulation method according to the current channel conditions.

[0052] At the same time, when performing quantization, for a network that has completed training, prior knowledge of the distribution of the data to be quantized is required. However, during the network training process, since the intermediate feature distribution will change dynamically, this method is difficult to apply during the model training process. And the current quantization methods only consider the quantization loss, without taking into account the task performance or the semantic importance of the parameters, which may lead to the ineffective protection of important semantic information, thus having an adverse impact on the task performance. At the same time, the quantization loss function is closely related to the number of interval segmentation points. When the number of segmentation points increases, the complexity of the loss function increases significantly, and the optimization difficulty increases, restricting the flexibility of adjusting the quantization level.

[0053] Based on the above description, this embodiment proposes a semantic information transmission method, as Figure 1 shown, which is applied to the sending end. The method includes:

[0054] Step 101: Receive the image to be detected, extract semantic features from the image to be detected, and obtain target semantic features;

[0055] Step 102: Perform encoding processing on the target semantic features to obtain a target semantic vector corresponding to the image to be detected;

[0056] Step 103: Input the target semantic vector into a pre-trained semantic importance discrimination model, and after being processed by the semantic importance discrimination model, output a target semantic importance value;

[0057] Step 104: Input the target semantic importance value into a pre-trained quantization level determination model, and after being processed by the quantization level determination model, output a target quantization level;

[0058] Step 105: Perform quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bit stream;

[0059] Step 106: Perform modulation processing on the target bit stream to obtain a target semantic signal, and send the target semantic signal and the target quantization level to the channel, and send them to the receiving end via the channel.

[0060] In specific implementation, the sending end receives the image to be detected. The sending end includes a feature extractor, and the feature extractor is used to extract semantic features from the image to be detected, obtaining target semantic features. The target semantic features are semantic features related to the detection task during object detection by the subsequent receiving end.

[0061] The sending end includes a channel encoder, and the channel encoder is used to perform encoding processing on the target semantic features, obtaining the target semantic vector corresponding to the image to be detected, which further improves the ability of the semantic features to resist channel noise interference.

[0062] The sending end includes a semantic importance discrimination module. The semantic importance discrimination module includes a semantic importance discrimination model, and the semantic importance discrimination model is a neural network model. The target semantic vector is input into the pre-trained semantic importance discrimination model, and after being processed by the semantic importance discrimination model, a target semantic importance value is output.

[0063] The sending end includes a quantization module. The quantization module includes a quantization level determination model. The target semantic importance value is input into the pre-trained quantization level determination model, and after being processed by the quantization level determination model, a target quantization level is output.

[0064] The sending end includes a bitstream generation module. The bitstream generation module receives the target quantization level output by the quantization module, and performs quantization encoding processing on the target semantic vector output by the channel encoder according to the target quantization level, obtaining a target bitstream.

[0065] The sending end also includes a modulation module. The modulation module receives the target bitstream generated by the bitstream generation module, modulates the target bitstream, obtaining a target semantic signal, and sends the target semantic signal to the channel.

[0066] In this embodiment, since the data volume of the quantization level is small, it can be used as control information and transmitted to the receiving end through the control channel.

[0067] In this embodiment, the channel simulates an AWGN channel. After adding noise to the target semantic signal, the noisy target semantic signal is transmitted to the receiving end.

[0068] Through the above solution, a to-be-detected image is received, semantic features of the to-be-detected image are extracted to obtain target semantic features. The target semantic features are encoded to obtain a target semantic vector corresponding to the to-be-detected image. The target semantic vector is input into a pre-trained semantic importance discrimination model, and through the processing of the semantic importance discrimination model, a target semantic importance value is output. The target semantic importance value is input into a pre-trained quantization level determination model, and through the processing of the quantization level determination model, a target quantization level is output. The corresponding target quantization level is determined by the semantic importance value, that is, subsequent non-uniform quantization of semantic information can be performed according to the semantic importance, improving the transmission efficiency of semantic information. The target semantic vector is quantized and encoded according to the target quantization level to obtain a target bitstream, and the target bitstream is modulated to obtain a target semantic signal. The target semantic signal and the target quantization level are sent to a channel and transmitted to a receiving end through the channel. By determining the corresponding quantization level according to the semantic importance value and performing quantization processing, it is avoided that when dividing according to high and low bit positions, the data in the low bit positions contains important semantic information, and it is also avoided that important semantic information is lost due to rule-based hard threshold truncation.

[0069] In some embodiments, the semantic importance discrimination model is pre-trained. Specifically, the training process of the semantic importance discrimination model includes:

[0070] Step 10A, obtaining a training image set and an initial semantic importance discrimination model, and determining a training semantic vector corresponding to each training image in the training image set;

[0071] Step 10B, using the training semantic vector to train the initial semantic importance discrimination model to obtain a training semantic importance value;

[0072] Step 10C, receiving a detection loss gradient sent by the receiving end, and determining a target loss function according to the training semantic vector, the training semantic importance value, and the detection loss gradient;

[0073] Step 10D, until the number of training times reaches a preset number threshold, determining that the training of the initial semantic importance discrimination model is completed to obtain a semantic importance discrimination model.

[0074] During specific implementation, a training image set and an initial semantic importance discrimination model are obtained, and a training semantic vector corresponding to each training image in the training image set is determined. For the training semantic vector corresponding to each training image, the determination method of the training semantic vector is specifically as follows:

[0075] Use a feature extractor to perform semantic feature extraction on the training images to obtain training semantic features. Encode the training semantic features to obtain the training semantic vectors corresponding to the training images.

[0076] In this embodiment, the semantic importance discrimination model is implemented based on depthwise separable convolution, which can simplify the network design and accelerate the calculation process of forward propagation. Depthwise separable convolution is divided into Figure 2 the two-step convolution shown in the figure. First, use grouped convolution to integrate the information within the channels, and then use 1x1 convolution to perform cross-channel information integration. Specifically, depthwise separable convolution separates the convolution process for space and channels. First, perform a separate convolution operation on each input channel to extract the features of each channel, and then merge the information of all channels through a 1x1 convolution kernel to achieve cross-channel information fusion. While ensuring that the model perceives spatial and channel information, it reduces the number of parameters and the amount of computation. Using depthwise separable convolution reduces the number of parameters by approximately 76% compared to general convolution.

[0077] Figure 3 The figure shows a schematic diagram of the basic building block of the semantic importance discrimination model in this embodiment. The basic building block is a depthwise separable convolution block, and instance normalization and the GELU activation function are used after each convolution. The semantic importance discrimination module is formed by connecting three basic building blocks in series. The overall input of the semantic importance discrimination module is the semantic information output by the channel encoder, that is, the training semantic vector described above. The judgment result, that is, the training semantic importance value described above, is output after the last layer of GELU.

[0078] Based on the design of depthwise separable convolution, in order to better assist the semantic importance discrimination module in discriminating the importance of semantic information, additional guidance is used for the training of the semantic importance discrimination model, thereby improving the accuracy of semantic importance judgment.

[0079] Specifically, for the object detection task, the object parts that need to detect the position and judge the category are important semantic information, while the background part is unimportant semantic information. When transmitting in a low signal-to-noise ratio channel, using a higher quantization level for important semantic information and a lower quantization level for unimportant semantic information can better protect the important semantic information and improve the performance of the object detection task.

[0080] Use the training semantic vectors to train the initial semantic importance discrimination model to obtain the training semantic importance values. Receive the detection loss gradient sent by the receiving end, and determine the target loss function according to the training semantic vectors, the training semantic importance values, and the detection loss gradient. The specific determination method of the target loss function includes:

[0081] Multiply the training semantic vector and the detection loss gradient by semantic elements to obtain an initial product value, where the training semantic vector includes at least one training semantic element.

[0082] Use a first activation function to normalize the initial product value to obtain a target product value;

[0083] Determine a first loss function according to the target product value and the training semantic importance value. The specific determination method of the first loss function is:

[0084] Calculate the cross-entropy between the target product value and the training semantic importance value to determine the first loss function, and the first loss function is the cross-entropy loss function. Among them, the first loss function is expressed by the formula:

[0085]

[0086] where D is the number of training semantic elements in the training semantic vector, i is the i-th semantic element, and sigmoid() is the first activation function.

[0087] Determine the training quantization level corresponding to the training semantic importance value, perform quantization encoding processing on the training semantic vector according to the training quantization level to obtain a training bitstream. Modulate the training bitstream to obtain a training semantic signal. Determine the initial number of bits corresponding to each training semantic element in the training semantic signal, and perform a mean process on all the initial numbers of bits to obtain a mean number of bits. Among them, the mean number of bits is expressed by the formula:

[0088]

[0089] where M1 is the mean number of bits, z i ′ is the training quantization level;

[0090] Obtain a preset transmission data volume threshold, calculate the absolute value between the mean number of bits and the preset transmission data volume mean, and determine a second loss function. Among them, the second loss function is expressed by the formula:

[0091] L Bit = |M1 - T|

[0092] where T is the preset transmission data volume threshold.

[0093] Perform an addition process according to the first loss function and the second loss function to obtain a target loss function. Among them, the target loss function is expressed by the formula:

[0094] L2 = L CrossEntropy(F1·G1,F2) + αL Bit

[0095] Wherein, L2 is the target loss function, and L CrossEntropy (F1·G1,F2) is the first loss function, and L Bit is the second loss function, α is the weight value corresponding to the second loss function, F1 is the training semantic vector, G1 is the detection loss gradient, and F2 is the training semantic importance value.

[0096] Specifically, as Figure 4 shown, taking the performance loss of the object detection task as the loss function, performing the first backpropagation to obtain F1, G1, and F2. The purpose of semantic importance guidance is to make the prediction F2 of the semantic importance discrimination module as close as possible to the product of the node value F1 and the corresponding gradient G1. Therefore, the cross-entropy between the element-wise product of F1 and G1 after normalization using sigmoid and F2 is used as one term in the loss function L2. The program will normalize F2 when calculating the cross-entropy, so there is no need to perform additional normalization on F2.

[0097] In addition, in order to limit the amount of data transmitted, the mean M1 of the number of bits corresponding to each semantic information element after quantization is controlled by the absolute value loss function to make it close to the required amount of transmitted data T. The weighted sum of the cross-entropy and the absolute value loss is used as L2, and the second backpropagation is performed to optimize the model parameters.

[0098] Continuously train the initial semantic importance discrimination model until the number of training times reaches the preset number threshold, determine that the training of the initial semantic importance discrimination model is completed, and obtain the semantic importance discrimination model.

[0099] Through the above solution, when training the semantic importance discrimination model, the node values and gradients of the network itself are used as the training guidance for the semantic importance discrimination module, reducing the computational amount of the network.

[0100] In some embodiments, step 104 specifically includes:

[0101] Step 1041: Input the target semantic importance value into the quantization level determination model, and calculate the probability distribution of the quantization levels of each target semantic element in the target semantic vector according to the target semantic importance value;

[0102] Step 1042: For the probability distribution of the quantization levels of each target semantic element, use the second activation function to determine the initial quantization level corresponding to the probability distribution, where the initial quantization level is represented by the formula:

[0103]

[0104] Among them, z i is the initial quantization level, τ is the hyperparameter temperature in the calculation of the second activation function, and g i is the result sampled from the Gumbel(0,1) distribution, and π i is the probability distribution.

[0105] Step 1043: Traverse all the initial quantization levels corresponding to the probability distributions, select the maximum value among the initial quantization levels as the target quantization level, and output the target quantization level.

[0106] In specific implementation, the quantization module includes a quantization level determination model, which is implemented based on a convolutional network. The quantization level determination model is pre-trained. The convolutional network contains convolutional kernels, and the size of the convolutional kernels in the convolutional network is 1x1. Input the target semantic importance value into the pre-trained quantization level determination model, and calculate the probability distribution of the quantization levels of each target semantic element in the target semantic vector according to the target semantic importance value.

[0107] As Figure 5 shown, Figure 5 shows a schematic diagram of the selection of quantization levels based on convolution in the present application. The semantic information importance of the input is on the left, and the probability distribution of quantization level selection is calculated by a 1x1 convolutional kernel according to the position. Figure 5 As shown in the output, the four channels respectively represent the selection probabilities of four quantization levels.

[0108] To achieve end-to-end training, the reparameterization technique is used to solve the gradient problem. As Figure 6 shown, Figure 6 shows the flowchart of the reparameterization described in this embodiment. The reparameterization method uses the following method to replace the sampling process during forward propagation. Specifically:

[0109] For the probability distribution of the quantization levels of each target semantic element, use the second activation function to determine the initial quantization level corresponding to the probability distribution, where the initial quantization level is represented by the formula:

[0110]

[0111] Among them, z i is the initial quantization level, τ is the hyperparameter temperature in the calculation of the second activation function, and g i is the result sampled from the Gumbel(0,1) distribution, and π i is the probability distribution of the quantization levels of the target semantic elements output by the convolutional network.

[0112] Traverse all the initial quantization levels corresponding to the probability distributions, and select the maximum value among the initial quantization levels as the target quantization level.

[0113] In this embodiment, since only the probability distribution of the quantization level of each target semantic element is related to the network parameters in the calculation, the gradient during the backpropagation process is Thus, the gradient problem in the sampling process is solved, and the end-to-end training is completed.

[0114] Through the above solution, the gradient problems of quantization and sampling operations are solved by the methods of straight-through gradient and reparameterization, and the end-to-end training with a quantization process is achieved.

[0115] In this embodiment, a lightweight network is used to discriminate the importance of semantic information, and more bits are allocated to the important semantic information accordingly. This method better protects the important semantic information compared to the division method based on bit positions, and improves the transmission efficiency of semantic information. At the same time, a unified modulation method can be adopted for the bitstream during transmission, without the need to select a modulation method.

[0116] At the same time, important semantic information is protected under the same quantization accuracy, and the range of selectable quantization levels can be changed only by adjusting the elements in the quantization level set, which improves the flexibility of selection and does not make the loss function difficult to optimize due to the increase in selectable quantization levels. At the same time, a neural network is used to dynamically select the quantization level according to the semantic importance, which is independent of the distribution of the data to be quantized, enabling the model to learn the impact of quantization on task performance during end-to-end training.

[0117] In this embodiment, the semantic importance of the output of the encoder is discriminated, and the quantization level is selected accordingly, rather than discriminating the semantic importance based on the original image. Such a design can obtain more accurate semantic importance, which is beneficial for the model to select the quantization level, thereby better protecting the important semantic information and improving the system performance. Important information is protected at the semantic level, avoiding the loss of important semantic information due to hard threshold truncation based on rules. At the same time, by introducing a separable convolutional network for semantic importance discrimination and quantization level selection, compared with the structure of a fully connected network, the computational complexity can be effectively reduced when facing complex information sources.

[0118] Based on the above description, this embodiment proposes a method for transmitting semantic information, as Figure 7 shown, applied to the receiving end, and the method includes:

[0119] Step 201, receive the target semantic signal and the target quantization level in the channel, and demodulate the target semantic signal to obtain a target bitstream;

[0120] Step 202: Perform inverse quantization processing on the target bitstream using the target quantization level to obtain a target semantic vector;

[0121] Step 203: Perform decoding processing on the target semantic vector to obtain a target semantic feature;

[0122] Step 204: Perform object detection on the image to be detected according to the target semantic feature to obtain the detection position and detection category of the target object.

[0123] In specific implementation, the receiving end includes a demodulation module that receives the target semantic signal and the target quantization level in the channel. The demodulation module performs demodulation processing on the target semantic signal to obtain a target bitstream. In this embodiment, hard decision is performed during demodulation, and after the demodulation processing is completed, the target bitstream is output.

[0124] The receiving end further includes an inverse quantization module and a channel decoder. The inverse quantization module receives the target bitstream output by the demodulation module, performs inverse quantization processing on the target bitstream to obtain a target semantic vector, and inputs the target semantic vector to the channel decoder.

[0125] The receiving end includes a detector. The channel decoder performs decoding processing on the target semantic vector to obtain a target semantic feature. The channel decoder inputs the target semantic feature to the detector, and the detector performs object detection on the image to be detected according to the target semantic feature to obtain the detection position and detection category of the target object.

[0126] Meanwhile, the actual position and actual category of the object in the image to be detected can be determined. According to the actual position, the actual category, the detection position, and the detection category, the detection loss gradient is determined, and then the detection loss gradient is sent to the sending end for the semantic importance discrimination model of the sending end to determine the target loss function during training. Since the gradient magnitude of the network nodes directly reflects their influence on the loss function during the backpropagation process, the larger the gradient, the greater the influence on the loss function and the performance of the detection task, and the corresponding higher semantic importance. Therefore, considering the detection loss gradient when determining the target loss function, the determined target loss function is more accurate.

[0127] Based on the same inventive concept, another embodiment of the present disclosure provides a semantic information transmission method, including a sending end, a channel, and a receiving end. The sending end includes a feature extractor, a channel encoder, a semantic importance discrimination module, a quantization module, and a modulation module. The quantization module includes a quantization level selection module and a bitstream generation module. The receiving end includes a demodulation module, an inverse quantization module, a channel decoder, and a detector.

[0128] In this embodiment, the semantic importance determination module includes the semantic importance determination model described in the above embodiment, and the quantization level selection module includes the quantization level determination model described in the above embodiment.

[0129] like Figure 8 As shown, Figure 8 A schematic diagram of a framework of a semantic information transmission method in this embodiment is shown, and the method includes:

[0130] The feature extractor first extracts semantic features from the image to be detected, obtains the target semantic features related to the detection task, and sends the extracted target semantic features to the channel encoder.

[0131] The channel encoder encodes the target semantic features output by the feature extractor to improve the ability of semantic information to resist channel noise interference and obtain the target semantic vector.

[0132] The semantic importance discrimination module performs semantic importance discrimination on the target semantic vector output by the channel encoder to obtain the importance of the semantic information, that is, outputs the target semantic importance value.

[0133] The quantization level selection module in the quantization module outputs the quantization level selection result corresponding to the semantic feature based on the target semantic importance value, obtaining the target quantization level. The bitstream generation module quantizes the target semantic vector output by the channel encoder according to the quantization level and encodes it into a target bitstream, which is then fed into the modulation module.

[0134] The modulation module modulates the received target bit stream to obtain the target semantic signal and sends it into the channel. The channel module simulates the AWGN channel to add noise to the target semantic signal and then transmits it to the receiving end.

[0135] The receiving end's demodulation module first demodulates the received target semantic signal, performs hard decision making, and generates the target bitstream, which is then fed into the channel decoder. The quantization level, which is small in size, can be used as control information and transmitted to the receiving end via the control channel. The modulation method used by the modulation module is not specified and can be adjusted based on actual conditions.

[0136] The dequantization module at the receiving end dequantizes the target bit stream according to the quantization level to obtain the target semantic vector and sends it to the channel decoder. The channel decoder processes the received target semantic vector to reduce the impact of channel noise, restore the semantic information, and obtain the target semantic features.

[0137] The detector uses the target semantic features output by the channel decoder to detect potential targets in the original image, gives the target location mark box and the detection result of the category to which it belongs, and completes the target detection task.

[0138] In this embodiment, the feature extractor and the detector are implemented based on a neural network. The lightweight network MobileNetV2 is used as the semantic feature extractor to generate a semantic feature map, and then the SingleShot MultiBox Detector (SSD) uses the semantic features extracted by the semantic feature extractor to complete the object detection task. The first 14 layers of MobileNetV2 are placed at the sending end to extract the semantic features of the information source.

[0139] The remaining part of MobileNetV2 is placed after the channel decoder at the receiving end, and the received semantic information is used to generate an additional-sized semantic feature map for object detection. SSD generates detection anchor boxes on the 6 different-sized feature maps generated by MobileNetV2, detects the positions and categories of objects of different sizes, and improves the detection performance by using the idea of multiple feature maps.

[0140] In this embodiment, the channel codec is implemented based on a neural network and is designed using the structure of an autoencoder. Specifically, the encoder further compresses and encodes the semantic information extracted by the feature extractor through a multi-layer convolutional network to improve the ability of the semantic information to resist channel noise interference. The decoder adopts a symmetric structure design with the encoder. Through the corresponding multi-layer transposed convolutional network, the influence of channel noise is reduced by using the correlation at the semantic level, and the semantic information extracted by the feature extractor is restored. The semantic information output by the channel decoder is sent to the detector for object detection.

[0141] In this embodiment, the semantic importance discrimination module is implemented based on a separable convolutional network. It receives the semantic features output by the channel encoder and outputs the result of discriminating the semantic importance of the semantic features after separable convolution.

[0142] To implement the quantization level selection based on semantic importance, the importance of the semantic features output by the channel encoder is discriminated, so that the model can select the quantization level based on semantic importance. The semantic importance discrimination module is implemented based on separable convolution, and separable convolution can simplify the network design and accelerate the calculation process of forward propagation.

[0143] Specifically, separable convolution is divided into two-step convolution, separating the convolution processes for space and channels. First, a separate convolution operation is performed on each input channel to extract the features of each channel, and then the information of all channels is merged through a 1x1 convolution kernel to achieve cross-channel information fusion. This can reduce the parameters and computational amount while ensuring that the model perceives the spatial and channel information. Using separable convolution reduces the number of parameters by about 76% compared with general convolution.

[0144] Based on the design of separable convolution, in order to better assist the semantic importance discrimination module in discriminating the importance of semantic information, additional guidance is used for the training of this module, thereby improving the accuracy of semantic importance judgment.

[0145] Specifically, for the object detection task, the object parts that need to detect positions and judge categories are important semantic information, while the background parts are unimportant semantic information. When transmitting in a low signal-to-noise ratio channel, using a higher quantization level for important semantic information and a lower quantization level for unimportant semantic information can better protect the important semantic information and improve the performance of the object detection task.

[0146] The product of the node value and the corresponding gradient is selected as the guidance for semantic information importance. Specifically, during the forward propagation process, network nodes with values very close to zero have little impact on the calculation of subsequent nodes, so they can hardly affect the output of the detector and the performance of the detection task, corresponding to lower semantic importance. At the same time, the magnitude of the gradient of network nodes during the backpropagation process directly reflects its impact on the loss function. The larger the gradient, the greater the impact on the loss function and the detection task performance, corresponding to higher semantic importance. Based on this, considering the influence of both on semantic importance, the product of the node value and the corresponding gradient is selected as the guidance for semantic information importance.

[0147] To implement the above process of guiding semantic importance and enable the semantic importance discrimination module to learn to discriminate semantic importance during end-to-end training, capture the semantic information F1 output by the channel encoder and its corresponding gradient G1 during backpropagation at the output position of the channel encoder, and at the same time capture the semantic importance discrimination result F2 at the output position of the importance discrimination module.

[0148] During training, first use the performance loss of the object detection task as the loss function and perform the first backpropagation to obtain F1, G1, and F2. The purpose of semantic importance guidance is to make the prediction F2 of the semantic importance discrimination module as close as possible to the product of the node value F1 and the corresponding gradient G1. Therefore, the element-wise product of F1 and G1 and the cross-entropy with F2 after sigmoid normalization are used as one term in the loss function L2. The program will normalize F2 when calculating the cross-entropy, so there is no need to normalize F2 additionally. In addition, in order to limit the amount of transmitted data, the mean M1 of the number of bits corresponding to each semantic information element after quantization is controlled by the absolute value loss function to make it close to the requirement T for the amount of transmitted data. The weighted sum of the cross-entropy and the absolute value loss is used as L2, and the second backpropagation is performed to optimize the model parameters.

[0149] In this embodiment, the loss function of the semantic importance discrimination module is expressed by the formula:

[0150] L2 = L CrossEntropy (F1·G1,F2) + αL Bit

[0151]

[0152]

[0153] Among them, L2 is the loss function, and L CrossEntropy (F1·G1,F2) is the first loss function, and L Bit is the second loss function, α is the weight value corresponding to the second loss function, F1 is the training semantic vector, G1 is the detection loss gradient, F2 is the training semantic importance value, D is the number of training semantic elements in the training semantic vector, i is the i-th semantic element, sigmoid() is the first activation function, M1 is the mean number of bits, z i ′ is the training quantization level, and T is the preset transmission data volume threshold.

[0154] The quantization module includes a quantization level selection module and a bitstream generation module. The quantization level selection module is implemented by a single-layer convolutional network. It receives the semantic importance discrimination result, and through the single-layer convolutional network, outputs the quantization level of the semantic information. The bitstream generation module quantizes and encodes the semantic information output by the channel encoder according to the quantization level to obtain the bitstream to be transmitted. The dequantization module performs the opposite operation of the quantization module, restores the received bitstream to a numerical value according to the quantization level, and sends it to the channel decoder for processing.

[0155] The quantization module includes a quantization level selection module and a bitstream generation module. The quantization level selection module is implemented by a single-layer convolutional network. It receives the semantic importance discrimination result, and through the single-layer convolutional network, outputs the quantization level of the semantic information. The bitstream generation module quantizes and encodes the semantic information output by the channel encoder according to the quantization level to obtain the bitstream to be transmitted. The dequantization module performs the opposite operation of the quantization module, restores the received bitstream to a numerical value according to the quantization level, and sends it to the channel decoder for processing.

[0156] In this embodiment, in order to select the quantization level for the semantic information according to the semantic importance, a quantization level selection module based on a neural network is designed. The quantization level selection module, as a part of the quantization module, provides the corresponding quantization level for the bit generation module, so that the model can complete the process of quantization level selection during the forward propagation process.

[0157] The quantization level selection module is implemented based on a convolutional network. Its input is the semantic importance output from the semantic importance discrimination module, and it outputs the quantization level of each element according to the pixel position. Specifically, the quantization level selection module first uses a convolution with a 1×1 convolution kernel to calculate the probability distribution of the quantization level that each element should select based on the semantic importance.

[0158] In the process of selecting the quantization level, the process of sampling according to the probability distribution cannot perform backpropagation. To achieve end-to-end training, a reparameterization technique is used to solve the gradient problem. The specific process is as follows:

[0159] For the probability distribution of the quantization level of each target semantic element, the second activation function is used to determine the initial quantization level corresponding to the probability distribution, where the initial quantization level is represented by the formula:

[0160]

[0161] where, z i is the initial quantization level, the optional quantization levels come from the quantization level set, g i represents the result sampled from the Gumbel(0,1) distribution, τ is the hyperparameter temperature in the softmax calculation, and the larger τ is, the closer the calculated z i is to the uniform distribution. When selecting τ, it is required that the sampled result does not deviate from the original distribution and does not affect the stability of model training. In this embodiment, τ = 1.

[0162] In this embodiment, the largest z i is selected as the quantization level selection result of the semantic information output by the channel encoder, that is, the target quantization level. Because during the calculation, only π i is related to the network parameters, so the gradient during the backpropagation process is Thus, the gradient problem during the sampling process is solved, and end-to-end training is completed.

[0163] In this embodiment, the modulation module modulates the bitstream generated by the quantization module and sends it into the channel. The channel module simulates an AWGN channel to add noise and controls the channel to transmit the quantization level. The demodulation module at the receiving end demodulates the signal and performs hard decision, and obtains the bitstream and sends it into the inverse quantization module. Since the gradient of the quantization operation is 0, it will block the gradient propagation of the subsequent module to the previous module, so end-to-end optimization cannot be completed. To solve the gradient problem, the straight-through gradient method as shown in Figure 9 is used to directly feedback the gradient G received from the subsequent module to the previous module, so that the model can perform end-to-end training with the quantization process.

[0164] In this embodiment, the quantization of the semantic information output by the channel encoder is completed by the semantic importance discrimination module and the quantization level selection module. After being transmitted through the channel, the target detection task is completed at the receiving end, realizing the quantization and transmission process of semantic information based on semantic importance. At the same time, a non-uniform quantization method based on semantic importance is proposed, and the neural network performs non-uniform quantization on the semantic information according to the semantic importance, improving the transmission efficiency of the semantic information.

[0165] In this embodiment, the node values and gradients of the network itself are used as the training guidance for the semantic importance discrimination module, reducing the computational amount of the network. At the same time, the quantization level selection module selects the corresponding quantization level for each element, and solves the gradient problem of the quantization and sampling operations through the straight-through gradient and reparameterization methods, realizing the end-to-end training with the quantization process.

[0166] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0167] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure further provides a semantic information transmission device disposed at the sending end.

[0169] Refer to Figure 10 , Figure 10 For the semantic information transmission device of the embodiment, including:

[0170] An image receiving module 301, configured to receive the image to be detected, extract semantic features from the image to be detected, and obtain target semantic features;

[0171] An encoding processing module 302, configured to perform encoding processing on the target semantic features to obtain a target semantic vector corresponding to the image to be detected;

[0172] The model processing module 303 is configured to input the target semantic vector into a pre-trained semantic importance discrimination model, and output a target semantic importance value after being processed by the semantic importance discrimination model;

[0173] The quantization level determination module 304 is configured to input the target semantic importance value into a pre-trained quantization level determination model, and output a target quantization level after being processed by the quantization level determination model;

[0174] The bitstream determination module 305 is configured to perform quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bitstream;

[0175] The data sending module 306 is configured to modulate the target bitstream to obtain a target semantic signal, and send the target semantic signal and the target quantization level to a channel, and send them to a receiving end via the channel.

[0176] In some embodiments, the device further includes a model training module, and the model training module is specifically configured to:

[0177] Obtain a training image set and an initial semantic importance discrimination model, and determine a training semantic vector corresponding to each training image in the training image set;

[0178] Use the training semantic vector to train the initial semantic importance discrimination model to obtain a training semantic importance value;

[0179] Receive the detection loss gradient sent by the receiving end, and determine a target loss function according to the training semantic vector, the training semantic importance value, and the detection loss gradient;

[0180] Until the number of training times reaches a preset number threshold, determine that the training of the initial semantic importance discrimination model is completed, and obtain a semantic importance discrimination model.

[0181] In some embodiments, the model training module is further specifically configured to:

[0182] Perform a product process on the training semantic vector and the detection loss gradient according to semantic elements to obtain an initial product value, where the training semantic vector includes at least one training semantic element;

[0183] Use a first activation function to normalize the initial product value to obtain a target product value;

[0184] Determine a first loss function according to the target product value and the training semantic importance value;

[0185] Quantize the training semantic vectors according to the training semantic importance values to obtain training semantic signals;

[0186] Determine the initial number of bits corresponding to each training semantic element in the training semantic signals, and determine the second loss function according to all the initial numbers of bits;

[0187] Perform an addition process according to the first loss function and the second loss function to obtain a target loss function, where the target loss function is represented by the formula:

[0188] L2 = L CrossEntropy (F1·G1,F2) + αL Bit

[0189] where L2 is the target loss function, L CrossEntropy (F1·G1,F2) is the first loss function, L Bit is the second loss function, α is the weight value corresponding to the second loss function, F1 is the training semantic vector, G1 is the detection loss gradient, and F2 is the training semantic importance value.

[0190] In some embodiments, the model training module is further specifically configured to:

[0191] Calculate the cross entropy between the target product value and the training semantic importance value, and determine the first loss function, where the first loss function is represented by the formula:

[0192]

[0193] where D is the number of training semantic elements in the training semantic vector, i is the i-th semantic element, and sigmoid() is the first activation function.

[0194] In some embodiments, the model training module is further specifically configured to:

[0195] Determine the training quantization levels corresponding to the training semantic importance values, and determine the initial number of bits corresponding to each training semantic element in the training semantic signals according to the training quantization levels;

[0196] Perform a mean process on all the initial numbers of bits to obtain a mean number of bits, where the mean number of bits is represented by the formula:

[0197]

[0198] where M1 is the mean number of bits, z i ′ is the training quantization level;

[0199] Obtain a preset transmission data volume threshold, calculate the absolute value between the mean number of bits and the preset transmission data volume mean, and determine a second loss function, where the second loss function is expressed by the formula:

[0200] L Bit = |M1 - T|

[0201] where T is the preset transmission data volume threshold.

[0202] In some embodiments, the quantization level determination module 304 is specifically configured to:

[0203] Input the target semantic importance value into the quantization level determination model;

[0204] Use the convolutional kernels in the quantization level determination model to calculate the probability distribution of the quantization levels of each target semantic element in the target semantic vector according to the target semantic importance value;

[0205] For the probability distribution of the quantization levels of each target semantic element, use a second activation function to determine the initial quantization level corresponding to the probability distribution, where the initial quantization level is expressed by the formula:

[0206]

[0207] where z i is the initial quantization level, τ is the hyperparameter temperature in the calculation of the second activation function, g i is the result sampled from the Gumbel(0,1) distribution, and π i is the probability distribution;

[0208] Traverse the initial quantization levels corresponding to all probability distributions, select the maximum value among the initial quantization levels as the target quantization level, and output the target quantization level.

[0209] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides a semantic information transmission device disposed at the receiving end.

[0210] Refer to Figure 11 Figure 11 For the semantic information transmission device of the embodiment, it includes:

[0211] A data receiving module 401, configured to receive the target semantic signal and the target quantization level in the channel, demodulate the target semantic signal, and obtain a target bit stream;

[0212] An inverse quantization processing module 402, configured to perform inverse quantization processing on the target bit stream using the target quantization level to obtain a target semantic vector; ​

[0213] The decoding processing module 403 is configured to perform decoding processing on the target semantic vector to obtain a target semantic feature;

[0214] The detection module 404 is configured to perform object detection on the image to be detected by using the target semantic feature, and obtain the detection position and detection category of the target object.

[0215] In some embodiments, the device further includes a loss gradient determination module, and the loss gradient determination module is specifically configured to:

[0216] Determine the actual position and actual category of the object in the image to be detected;

[0217] Determine the detection loss gradient according to the actual position, the actual category, the detection position, and the detection category.

[0218] For convenience of description, when describing the above device, various modules are described separately according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0219] The device in the above embodiment is used to implement the corresponding semantic information transmission method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again.

[0220] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the semantic information transmission method described in any of the above embodiments is implemented.

[0221] Figure 12 FIG. shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0222] The processor 1010 may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0223] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store the operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and called and executed by the processor 1010.

[0224] The input / output interface 1030 is used to connect to the input / output module to achieve information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input devices can include keyboards, mice, touchscreens, microphones, various sensors, etc., and the output devices can include displays, speakers, vibrators, indicator lights, etc.

[0225] The communication interface 1040 is used to connect to the communication module (not shown in the figure) to achieve communication and interaction between this device and other devices. Among them, the communication module can achieve communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0226] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0227] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, this device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and do not necessarily include all the components shown in the figure.

[0228] The electronic device in the above embodiment is used to implement the corresponding semantic information transmission method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0229] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the semantic information transmission method as described in any of the foregoing embodiments.

[0230] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0231] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the semantic information transmission method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0232] It can be understood that before using the technical solutions of the various embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0233] For example, in response to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application program, server, or storage medium that performs the operations of the technical solutions of the present disclosure according to the prompt message.

[0234] As an optional but non-limiting implementation manner, the way of sending a prompt message to the user in response to receiving an active request from the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0235] It can be understood that the above process of notifying and obtaining the user's authorization is only illustrative and does not constitute a limitation on the implementation manner of the present disclosure. Other ways that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0236] Those of ordinary skill in the art should understand that: The discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; Under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, and they are not provided in detail for the sake of brevity.

[0237] In addition, for simplicity of explanation and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0238] Although the present disclosure has been described in connection with specific embodiments of the present disclosure, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0239] The embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A semantic information transmission method, characterized in that Applied to the sending end, including: Receiving the image to be detected, extracting semantic features from the image to be detected to obtain target semantic features; Performing encoding processing on the target semantic features to obtain a target semantic vector corresponding to the image to be detected; Inputting the target semantic vector into a pre-trained semantic importance discrimination model, and after being processed by the semantic importance discrimination model, outputting a target semantic importance value; Inputting the target semantic importance value into a pre-trained quantization level determination model, and after being processed by the quantization level determination model, outputting a target quantization level; Performing quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bitstream; Performing modulation processing on the target bitstream to obtain a target semantic signal, and sending the target semantic signal and the target quantization level to a channel and sending them to the receiving end via the channel.

2. The method according to claim 1, characterized in that, The training process of the semantic importance discrimination model includes: Obtaining a training image set and an initial semantic importance discrimination model, and determining a training semantic vector corresponding to each training image in the training image set; Training the initial semantic importance discrimination model by using the training semantic vector to obtain a training semantic importance value; Receiving the detection loss gradient sent by the receiving end, and determining a target loss function according to the training semantic vector, the training semantic importance value and the detection loss gradient; Until the number of training times reaches a preset number threshold, determining that the training of the initial semantic importance discrimination model is completed to obtain a semantic importance discrimination model.

3. The method according to claim 2, characterized in that, The determining the target loss function according to the training semantic vector, the training semantic importance value and the detection loss gradient includes: Performing a product process on the training semantic vector and the detection loss gradient according to semantic elements to obtain an initial product value, where the training semantic vector includes at least one training semantic element; Performing normalization processing on the initial product value by using a first activation function to obtain a target product value; Determining a first loss function according to the target product value and the training semantic importance value; Performing quantization processing on the training semantic vector according to the training semantic importance value to obtain a training semantic signal; Determining an initial number of bits corresponding to each training semantic element in the training semantic signal, and determining a second loss function according to all the initial numbers of bits; Performing an addition process on the first loss function and the second loss function to obtain a target loss function, where the target loss function is represented by a formula as: L2 = L CrossEntropy (F1·G1,F2) + αL Bit Among them, L2 is the target loss function, L CrossEntropy (F1·G1,F2) is the first loss function, L Bit is the second loss function, α is the weight value corresponding to the second loss function, F1 is the training semantic vector, G1 is the detection loss gradient, and F2 is the training semantic importance value.

4. The method according to claim 3, characterized in that, The determining the first loss function according to the target product value and the training semantic importance value includes: Calculating the cross entropy between the target product value and the training semantic importance value to determine a first loss function, where the first loss function is represented by a formula as: Where D is the number of training semantic elements in the training semantic vector, i is the i-th semantic element, and sigmoid() is the first activation function.

5. The method according to claim 3, wherein The determining the initial number of bits corresponding to each training semantic element in the training semantic signal and determining the second loss function according to all the initial numbers of bits includes: Determine the training quantization level corresponding to the training semantic importance value, and determine the initial number of bits corresponding to each training semantic element in the training semantic signal according to the training quantization level; Perform mean processing on all the initial numbers of bits to obtain a mean number of bits, where the mean number of bits is represented by the formula: where M1 is the mean number of bits, and z i ′ is the training quantization level; Obtain a preset transmission data volume threshold, calculate the absolute value between the mean number of bits and the preset transmission data volume mean, and determine a second loss function, where the second loss function is represented by the formula: L Bit = |M1 - T| where T is the preset transmission data volume threshold.

6. The method according to claim 1, wherein The step of inputting the target semantic importance value into a pre-trained quantization level determination model and outputting a target quantization level after being processed by the quantization level determination model includes: Input the target semantic importance value into the quantization level determination model, and calculate the probability distribution of the quantization level of each target semantic element in the target semantic vector according to the target semantic importance value; For the probability distribution of the quantization level of each target semantic element, use a second activation function to determine the initial quantization level corresponding to the probability distribution, where the initial quantization level is represented by the formula: where z i is the initial quantization level, τ is the hyperparameter temperature in the second activation function calculation, g i is the result sampled from the Gumbel(0,1) distribution, and π i is the probability distribution; Traverse the initial quantization levels corresponding to all the probability distributions, select the maximum value among the initial quantization levels as the target quantization level, and output the target quantization level.

7. A semantic information transmission method, characterized in that, When applied to the receiving end, it includes: Receive the target semantic signal and the target quantization level in the channel, perform demodulation processing on the target semantic signal to obtain a target bitstream; Perform inverse quantization processing on the target bitstream using the target quantization level to obtain a target semantic vector; Perform decoding processing on the target semantic vector to obtain a target semantic feature; Perform object detection on the image to be detected according to the target semantic feature to obtain the detection position and detection category of the target object.

8. The method according to claim 7, wherein After obtaining the detection position and detection category of the target object, it further includes: Determine the actual position and actual category of the object in the image to be detected; Determine the detection loss gradient according to the actual position, the actual category, the detection position and the detection category.

9. A semantic information transmission device, characterized in that, When set at the sending end, it includes: An image receiving module configured to receive the image to be detected, perform semantic feature extraction on the image to be detected to obtain a target semantic feature; An encoding processing module configured to perform encoding processing on the target semantic feature to obtain a target semantic vector corresponding to the image to be detected; A model processing module configured to input the target semantic vector into a pre-trained semantic importance discrimination model, and output a target semantic importance value after being processed by the semantic importance discrimination model; A quantization level determination module configured to input the target semantic importance value into a pre-trained quantization level determination model, and output a target quantization level after being processed by the quantization level determination model; A bitstream determination module configured to perform quantization encoding processing on the target semantic vector according to the target quantization level to obtain a target bitstream; A data sending module, which is configured to perform modulation processing on the target bitstream to obtain a target semantic signal, and send the target semantic signal and the target quantization level to a channel, and send them to a receiving end via the channel.

10. A semantic information transmission device, characterized in that Set at the receiving end, including: A data receiving module, which is configured to receive the target semantic signal and the target quantization level in the channel, and perform demodulation processing on the target semantic signal to obtain a target bitstream; An inverse quantization processing module, which is configured to perform inverse quantization processing on the target bitstream by using the target quantization level to obtain a target semantic vector; A decoding processing module, which is configured to perform decoding processing on the target semantic vector to obtain a target semantic feature; A detection module, which is configured to perform object detection on the image to be detected by using the target semantic feature to obtain the detection position and detection category of the target object.

Images